Image-capturing device

ABSTRACT

An image-capturing device includes an image-capturing unit  30  including a first image-capturing element  41 , a second image-capturing element  51 , a third image-capturing element  61 , and a fourth image-capturing element  71 , and includes an image processing unit  11 , wherein a sensitivity of the fourth image-capturing element  71  is less than sensitivities of the first image-capturing element  41  to the third image-capturing element  61 , and the image processing unit  11  generates high sensitivity image data on the basis of outputs from the first image-capturing element  41  to the third image-capturing element  61 , and generates low sensitivity image data on the basis of an output from the fourth image-capturing element  71 , and further, the image processing unit  11  generates a combined image using high sensitivity image data corresponding to a low illumination image area in the low illumination image area obtained from the low sensitivity image data or the high sensitivity image data, and using low sensitivity image data corresponding to an high illumination image area.

TECHNICAL FIELD

The present disclosure relates to an image-capturing device, and morespecifically, relates to an image-capturing device for obtaining animage by high dynamic range (HDR) imaging.

BACKGROUND ART

In an image-capturing device in which image-capturing elements arearranged in a two-dimensional matrix form, a technique for obtainingmultiple pieces of image data by performing image-capturing processmultiple times and combining the multiple pieces of image data is wellknown as indicated in, for example, JP 2000-244797 A. Alternatively, inorder to cope with an image-capturing scene having great difference inthe brightness, N pieces of image data of which sensitivities aredifferent are obtained by performing image-capturing process of a stillpicture N times (N is equal to or larger than 2) with different exposuretimes, and an HDR-combined image is obtained by combining these N piecesof image data. However, in the method for obtaining these N pieces ofimage data, it takes a certain period of time to obtain the N pieces ofimage data, and the simultaneousness cannot be ensured, and in addition,artifact occurs due to the combining of the N pieces of image data, andit is difficult to cope with motion pictures.

A technique for arranging image-capturing elements of differentsensitivities adjacent to each other, and expanding the dynamic range bycombining a signal from an image-capturing element of a highersensitivity and a signal from an image-capturing element of a lowersensitivity is well known as indicated in JP 2006-270364 A. In atechnique disclosed in JP 2006-253876 A, the difference in thesensitivity is given within one piece of image data by performingcontrol so as to change the electrical-charge accumulation time of eachimage-capturing element for each area of the image-capturing unit bycontrolling the shutter speed.

CITATION LIST Patent Documents

-   Patent Document 1: JP 2000-244797 A-   Patent Document 2: JP 2006-270364 A-   Patent Document 3: JP 2006-253876 A

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

In JP 2006-270364 A, a small image-capturing element having acomplementary filter is arranged in an image-capturing element arrangedin a honeycomb form, so that two pieces of image data of whichsensitivities are different can be obtained in a unit frame. In thismethod, however, due to the reduction of the size of the entireimage-capturing element, it is difficult to form a small image-capturingelement having a complementary filter, and the arrangement of the colorfilter is complicated. In addition, there is a problem of increase ofthe color filter materials and the process cost. The control of theshutter speed for each area in the technique disclosed in JP 2006-253876A increases the complexity of the control circuit and the controlalgorithm of the image-capturing device, and as a result, the costincreases. Further, the techniques disclosed in these patentpublications explained above increases the complexity of the structureof the image-capturing device and increases the cost of image processingsuch as resolution recovery, solving of jaggy, and color noisecompensation.

Therefore, it is an object of the present disclosure to provide animage-capturing device having a simple configuration and structurecapable of obtaining an image by high dynamic range imaging, and inaddition, capable of easily coping with image-capturing of a motionpicture.

Solutions to Problems

An image-capturing device according to the first aspect of the presentdisclosure for achieving the object explained above is animage-capturing device including: an image-capturing unit; and an imageprocessing unit, wherein the image-capturing unit includesimage-capturing element units arranged in a two-dimensional matrix form,and each of the image-capturing element units includes a firstimage-capturing element including a first filter and a firstphotoelectric conversion element, and configured to receive light in afirst wavelength band; a second image-capturing element including asecond filter and a second photoelectric conversion element, andconfigured to receive light in a second wavelength band having a peakwavelength longer than a peak wavelength of the first wavelength band; athird image-capturing element including a third filter and a thirdphotoelectric conversion element, and configured to receive light in athird wavelength band having a peak wavelength longer than a peakwavelength of the second wavelength band; and a fourth image-capturingelement including a fourth filter and a fourth photoelectric conversionelement, and configured to receive light in the first wavelength band,the second wavelength band, and the third wavelength band, and wherein alight transmission rate of the fourth filter is less than a lighttransmission rate of the first filter, a light transmission rate of thesecond filter, and a light transmission rate of the third filter, andwherein the image processing unit generates high sensitivity image databased on outputs from the first image-capturing element, the secondimage-capturing element, and the third image-capturing element, andgenerates low sensitivity image data based on an output from the fourthimage-capturing element, and the image processing unit further generatesa combined image using high sensitivity image data corresponding to alow illumination image area in the low illumination image area obtainedfrom the low sensitivity image data or the high sensitivity image data,and using low sensitivity image data corresponding to an highillumination image area in the high illumination image area obtainedfrom the low sensitivity image data or the high sensitivity image data.With the image-capturing device according to the first aspect of thepresent disclosure, the combined image can be obtained, in which aportion of the image is a single-color image, and the remaining portionis a color image.

An image-capturing device according to the second aspect of the presentdisclosure for achieving the object explained above is animage-capturing device including: an image-capturing unit; and an imageprocessing unit, wherein the image-capturing unit includesimage-capturing element units arranged in a two-dimensional matrix form,and each of the image-capturing element units includes: a firstimage-capturing element and a third image-capturing element including afirst photoelectric conversion element and configured to receive lightin a visible light range; and a second image-capturing element and afourth image-capturing element including a neutral density filter and asecond photoelectric conversion element and configured to receive lightin a visible light range, wherein the image processing unit generateshigh sensitivity image data on the basis of outputs from the firstimage-capturing element and the third image-capturing element, andgenerates low sensitivity image data on the basis of outputs from thesecond image-capturing element and the fourth image-capturing element,and the image processing unit further generates a combined image usinghigh sensitivity image data corresponding to a low illumination imagearea in the low illumination image area obtained from the lowsensitivity image data or the high sensitivity image data and using lowsensitivity image data corresponding to a high illumination image areain the high illumination image area obtained from the low sensitivityimage data or the high sensitivity image data. With the image-capturingdevice according to the second aspect of the present disclosure, asingle-color image can be obtained.

An image-capturing device according to the third aspect of the presentdisclosure for achieving the object explained above is animage-capturing device including: an image-capturing unit; and an imageprocessing unit, wherein the image-capturing unit includesimage-capturing element units arranged in a two-dimensional matrix form,and each of the image-capturing element units includes fourimage-capturing element sub-units each including a first image-capturingelement including a first filter and a first photoelectric conversionelement and configured to receive light in a first wavelength band; asecond image-capturing element including a second filter and a secondphotoelectric conversion element and configured to receive light in asecond wavelength band having a peak wavelength longer than a peakwavelength of the first wavelength band; a third image-capturing elementincluding a third filter and a third photoelectric conversion elementand configured to receive light in a third wavelength band having a peakwavelength longer than a peak wavelength of the second wavelength band;and a fourth image-capturing element including a fourth filter and afourth photoelectric conversion element and configured to receive lightin the first wavelength band, the second wavelength band, and the thirdwavelength band, wherein a light transmission rate of the fourth filteris less than a light transmission rate of the first filter, a lighttransmission rate of the second filter, and a light transmission rate ofthe third filter, and the image processing unit generates highsensitivity image data on the basis of an output summation of the firstimage-capturing elements, an output summation of the secondimage-capturing elements, and an output summation of the thirdimage-capturing elements of the four image-capturing element sub-unitsconstituting the image-capturing element unit, and generates lowsensitivity image data on the basis of an output from the fourthimage-capturing element of each of the four image-capturing elementsub-units, and the image processing unit further generates a combinedimage using high sensitivity image data corresponding to a lowillumination image area in the low illumination image area obtained fromthe low sensitivity image data or the high sensitivity image data, andusing low sensitivity image data corresponding to an high illuminationimage area in the high illumination image area obtained from the lowsensitivity image data or the high sensitivity image data. With theimage-capturing device according to the third aspect of the presentdisclosure, the combined image can be obtained, in which a portion ofthe image is a single-color image, and the remaining portion is a colorimage.

An image-capturing device according to the fourth aspect of the presentdisclosure for achieving the object explained above is animage-capturing device that includes an image-capturing unit and animage processing unit, wherein the image-capturing unit is arranged withimage-capturing element units, and each of the image-capturing elementunits includes: a first image-capturing element including a firstphotoelectric conversion element and configured to receive light in avisible light range and a second image-capturing element including aneutral density filter and a second photoelectric conversion element andconfigured to receive light in a visible light range. The imageprocessing unit generates high sensitivity image data on the basis of anoutput from the first image-capturing element and generates lowsensitivity image data on the basis of an output from the secondimage-capturing element, and the image processing unit further generatesa combined image using high sensitivity image data corresponding to alow illumination image area in the low illumination image area obtainedfrom the low sensitivity image data or the high sensitivity image dataand using low sensitivity image data corresponding to a highillumination image area in the high illumination image area obtainedfrom the low sensitivity image data or the high sensitivity image data.With the image-capturing device according to the fourth aspect of thepresent disclosure, a single-color image can be obtained.

Effects of the Invention

Depending on application fields and usage fields of an image-capturingdevice, color information is not necessarily required in a highillumination portion of an image-capturing scene in which there is greatdifference in the brightness, and even image acquisition of a gray imagewithout so-called blown out highlights would not cause any problem inpractical point of view. Examples of such fields include a monitoringcamera and an on-board camera which are strongly required “not to loseany information.” An image-capturing device according to the presentdisclosure basically generates low sensitivity image data and highsensitivity image data on the basis of an output from an image-capturingunit in one image-capturing process, and an image processing unit usesthe high sensitivity image data in a low illumination image area anduses the low sensitivity image data in a high illumination image area togenerate a combined image. Therefore, although the image-capturingdevice has, for example, the simple configuration and structureequivalent to those of the image-capturing device having a conventionalBayer arrangement, the image-capturing device can easily obtain an imagebased on a high dynamic range imaging, and can easily cope withimage-capturing of a motion picture. For this reason, a low-costimage-capturing device based on high dynamic range imaging method can beprovided.

BRIEF DESCRIPTION OF DRAWINGS

FIGS. 1A and 1B are a conceptual diagram illustrating an image-capturingunit of an image-capturing device of the first embodiment, and a figureschematically illustrating light reception characteristics of a firstimage-capturing element, a second image-capturing element, a thirdimage-capturing element, and a fourth image-capturing element,respectively.

FIGS. 2A and 2B are conceptual diagrams for explaining operation of animage-capturing unit of the image-capturing device of the firstembodiment.

FIGS. 3A and 3B are conceptual diagrams for explaining operation of theimage-capturing unit of the image-capturing device of the firstembodiment.

FIG. 4 is a conceptual diagram for explaining operation of theimage-capturing unit of the image-capturing device of the firstembodiment.

FIG. 5 is a conceptual diagram for explaining operation of theimage-capturing unit of the image-capturing device of the firstembodiment.

FIG. 6 is a figure schematically illustrating an image-capturing stateof the image-capturing device of the first embodiment.

FIGS. 7A and 7B are a conceptual diagram of the image-capturing deviceof the first embodiment, and a schematic partial cross sectional viewillustrating an image-capturing element, respectively.

FIG. 8 is a schematic partial top view illustrating a light shield layerwhen an image-capturing element is cut along arrow B-B of FIG. 7B in theimage-capturing device of the second embodiment.

FIGS. 9A and 9B are a figure illustrating a sensitivity ratio of thefirst image-capturing element, the second image-capturing element, thethird image-capturing element, and the fourth image-capturing element ofthe image-capturing device of the third embodiment, and a figureillustrating a gain ratio of the first image-capturing element, thesecond image-capturing element, the third image-capturing element, andthe fourth image-capturing element thereof, respectively.

FIGS. 10A and 10B are a final output ratio of high sensitivity imagedata of the image-capturing device of the third embodiment, and a figureillustrating a final output ratio of low sensitivity image data,respectively.

FIGS. 11A and 11B are a figure schematically illustrating animage-capturing state (exposure time) of the image-capturing device ofthe fourth embodiment in accordance with elapse of time, and a figureschematically illustrating output of the first image-capturing element,the second image-capturing element, the third image-capturing element,and the fourth image-capturing element, respectively.

FIG. 12 is a schematic partial top view illustrating an exposure maskwith a corner serif pattern for forming a filter on the basis of alithography technique.

FIGS. 13A and 13B are schematic partial top views of filters forexplaining a method for forming a first filter, a second filter, a thirdfilter, and a fourth filter according to the fifth embodiment.

FIGS. 14A and 14B are, continued from FIG. 13B, schematic partial topviews of the filters for explaining the method for forming the firstfilter, the second filter, the third filter, and the fourth filteraccording to the fifth embodiment.

FIGS. 15A and 15B are schematic partial top views of filters forexplaining another method for forming the first filter, the secondfilter, the third filter, and the fourth filter according to the fifthembodiment.

FIG. 16A is, continued from FIG. 15B, a schematic partial top view ofthe filters for explaining the another method for forming the firstfilter, the second filter, the third filter, and the fourth filteraccording to the fifth embodiment, and FIG. 16B is a schematic partialtop view of the second filter and the like for explaining still anothermethod for forming the first filter, the second filter, the thirdfilter, and the fourth filter according to the fifth embodiment.

FIGS. 17A and 17B are a conceptual diagram illustrating animage-capturing unit of an image-capturing device of the sixthembodiment, and a figure schematically illustrating light receptioncharacteristics of a first image-capturing element and a thirdimage-capturing element and of a second image-capturing element and afourth image-capturing element, respectively.

FIGS. 18A and 18B are conceptual diagrams for explaining operation ofthe image-capturing unit of the image-capturing device of the sixthembodiment.

FIGS. 19A and 19B are conceptual diagrams for explaining operation ofthe image-capturing unit of the image-capturing device of the sixthembodiment.

FIG. 20 is a conceptual diagram for explaining operation of theimage-capturing unit of the image-capturing device of the sixthembodiment.

FIG. 21 is a conceptual diagram for explaining operation of theimage-capturing unit of the image-capturing device of the sixthembodiment.

FIG. 22 is a schematic partial top view illustrating a light shieldlayer when an image-capturing element is cut along arrow B-B of FIG. 7Bin the image-capturing device of the seventh embodiment.

FIGS. 23A and 23B are a figure illustrating a sensitivity ratio of afirst image-capturing element, a second image-capturing element, a thirdimage-capturing element, and a fourth image-capturing element of theimage-capturing device of the seventh embodiment, and a figureillustrating a gain ratio caused by difference in the sizes of aperturesof the first image-capturing element, the second image-capturingelement, the third image-capturing element, and the fourthimage-capturing element thereof, respectively.

FIGS. 24A and 24B are figures illustrating final output ratios of highsensitivity image data according to the image-capturing device of theseventh embodiment.

FIGS. 25A and 25B are figures illustrating final output ratios of lowsensitivity image data according to the image-capturing device of theseventh embodiment.

FIG. 26 is a conceptual diagram illustrating an image-capturing unit ofan image-capturing device according to an eleventh embodiment.

FIGS. 27A and 27B are conceptual diagrams for explaining operation ofthe image-capturing unit of the image-capturing device according to theeleventh embodiment.

FIGS. 28A and 28B are conceptual diagrams for explaining operation ofthe image-capturing unit of the image-capturing device according to theeleventh embodiment.

FIG. 29 is a conceptual diagram for explaining operation of theimage-capturing unit of the image-capturing device according to theeleventh embodiment.

MODE FOR CARRYING OUT THE INVENTION

The present disclosure will be explained on the basis of embodimentswith reference to drawings, but the present disclosure is not limited tothe embodiments, and various numerical values and materials in theembodiments are merely examples. Explanation will be made in thefollowing order.

1. General explanation about image-capturing device according to firstaspect to fourth aspect of the present disclosure2. First embodiment (image-capturing device according to the firstaspect of the present disclosure)3. Second embodiment (modification of first embodiment)4. Third embodiment (another modification of first embodiment)5. Fourth embodiment (still another modification of first embodiment)6. Fifth embodiment (modification of first embodiment to fourthembodiment7. Sixth embodiment (image-capturing device according to second aspectand fourth aspect of the present disclosure8. Seventh embodiment (modification of sixth embodiment)9. Eighth embodiment (another modification of sixth embodiment)10. Ninth embodiment (still another modification of sixth embodiment)11. Tenth embodiment (modification of first embodiment and sixthembodiment)12. Eleventh embodiment (image-capturing device according to the thirdaspect of the present disclosure), and others

[General Explanation about Image-Capturing Device According to FirstAspect to Fourth Aspect of the Present Disclosure]

In an image-capturing device according to the first aspect of thepresent disclosure, a first aperture portion may be formed between afirst filter and a first photoelectric conversion element; a secondaperture portion may be formed between a second filter and a secondphotoelectric conversion element; a third aperture portion may be formedbetween a third filter and a third photoelectric conversion element; anda fourth aperture portion may be formed between a fourth filter and afourth photoelectric conversion element, wherein the fourth apertureportion may be smaller than the first aperture portion, the secondaperture portion, and the third aperture portion. With suchconfiguration, the quantity of light received by the fourthimage-capturing element is less than the quantity of light received bythe first image-capturing element, the second image-capturing element,and the third image-capturing element, and as a result, the dynamicrange can be expanded. An aperture portion can be obtained by formingthe aperture portion in a light shield layer formed between a filter anda photoelectric conversion element. The planar shape of the apertureportion may be a circular shape, or a regular polygonal shape (forexample, a regular pentagon, a regular hexagon, a regular heptagon, anda regular octagon). It should be noted that the regular polygonal shapeincludes quasi-regular polygonal shapes (a regular polygonal shape ofwhich sides are constituted by curved lines, and a regular polygonalshape of which vertexes are round). This is also applicable to animage-capturing device according to the second aspect of the presentdisclosure explained subsequently.

In an image-capturing device according to the second aspect of thepresent disclosure, the first aperture portion may be formed in a lightincidence area of the first photoelectric conversion element; the secondaperture portion may be formed between a neutral density filter and thesecond photoelectric conversion element; a third aperture portion may beformed in a light incidence area of the third photoelectric conversionelement; a fourth aperture portion may be formed between the neutraldensity filter and the fourth photoelectric conversion element, whereinthe third aperture portion may be smaller than the first apertureportion, and the fourth aperture portion may be smaller than the secondaperture portion. With such configuration, the quantity of lightreceived by the first image-capturing element is less than the quantityof light received by the third image-capturing element. In addition, thequantity of light received by the second image-capturing element is lessthan the quantity of light received by the fourth image-capturingelement. As a result, the dynamic range can be expanded. An apertureportion can be obtained by forming the aperture portion in a lightshield layer formed between a neutral density filter and a photoelectricconversion element.

In the image-capturing device according to the first aspect of thepresent disclosure, or the image-capturing device according to the thirdaspect of the present disclosure including the above preferableconfigurations, the image processing unit may include a first gainadjustment unit configured to adjust outputs from the firstimage-capturing element, the second image-capturing element, and thethird image-capturing element, and a second gain adjustment unitconfigured to adjust an output from the fourth image-capturing element.In this case, where an adjustment coefficient for the outputs from thefirst image-capturing element, the second image-capturing element, andthe third image-capturing element by the first gain adjustment unit isdenoted as Gn₁, and an adjustment coefficient for the output from thefourth image-capturing element by the second gain adjustment unit isdenoted as Gn₂, it is preferable to satisfy Gn₁/Gn₂≧1. Morespecifically, examples of values of (Gn₁/Gn₂) include 2, 4, 8, and thelike. Even with such configuration, the dynamic range can be expanded.The value of Gn₁/Gn₂ may be fixed at “1” in a case where there is asmall degree of difference in the brightness in an image-capturingscene, but in a case where there is great difference in the brightnessin an image-capturing scene, the value of Gn₁/Gn₂ may be automaticallyor manually changed in accordance with the difference between thebrightness of the high sensitivity image data in a low illumination areain the image-capturing scene and the brightness of the low sensitivityimage data in a high illumination area in the image-capturing scene.This is also applicable to an image-capturing device according to thesecond aspect of the present disclosure explained subsequently. Theadjustments of the outputs from the first image-capturing element, thesecond image-capturing element, and the third image-capturing element bythe first gain adjustment unit may be the same, or may be differentdepending on cases.

In the image-capturing device according to the second aspect of thepresent disclosure including the above preferable configuration, theimage processing unit may include a first gain adjustment unitconfigured to adjust outputs from the first image-capturing element andthe third image-capturing element, and a second gain adjustment unitconfigured to adjust outputs from the second image-capturing element andthe fourth image-capturing element. In this case, where an adjustmentcoefficient for outputs from the first image-capturing element and thethird image-capturing element by the first gain adjustment unit isdenoted as Gn₁, and an adjustment coefficient for outputs from thesecond image-capturing element and the fourth image-capturing element bythe second gain adjustment unit is denoted as Gn₂, it is preferable tosatisfy Gn₁/Gn₂≧1. More specifically, examples of values of (Gn₁/Gn₂)include 2, 4, 8, and the like. Even with such configuration, the dynamicrange can be expanded. The adjustments of the outputs from the firstimage-capturing element and the third image-capturing element by thefirst gain adjustment unit may be the same, or may be differentdepending on cases. The adjustments of the outputs from the secondimage-capturing element and the fourth image-capturing element by thesecond gain adjustment unit may be the same, or may be differentdepending on cases.

In the image-capturing device according to the first aspect or the thirdaspect of the present disclosure including the various kinds ofpreferable configurations explained above, the image processing unit maygenerate N sets of high sensitivity image data and low sensitivity imagedata of which exposure times are different, and the image processingunit may further divide an illumination area of an image obtained fromthe low sensitivity image data or the high sensitivity image data into2N levels of areas ranging from an area of which illumination is thelowest to an area of which illumination is the highest, and may generatea combined image using N sets of high sensitivity image datarespectively corresponding to N levels of low illumination image areasin the N levels of low illumination image areas ranging from the area ofwhich illumination is the lowest to the area of which illumination isthe N-th lowest, and using N sets of low sensitivity image datarespectively corresponding to N levels of high illumination image areasin the N levels of high illumination image areas ranging from the areaof which illumination is the (N+1)-th lowest to the area of whichillumination is the highest. In this case, N is 2, and there can be suchrelationship that the image-capturing time for obtaining the highsensitivity image data and the low sensitivity image data of the firstset is twice the image-capturing time for obtaining the high sensitivityimage data and the low sensitivity image data of the second set. Itshould be noted that such configuration can also be expanded to a casewhere N is equal to or more than 3. Even with such configuration, thedynamic range can be expanded. In addition, an image of which dynamicrange is expanded to 2N levels from N sets of image data can beobtained, and it is easy to cope with image-capturing of a motionpicture. In the conventional technique, as described above, only animage of which dynamic range is expanded to N levels from N sets ofimage data can be obtained.

In the image-capturing device according to the second aspect of thepresent disclosure including the various kinds of preferableconfigurations explained above, the image processing unit may generate Nsets of high sensitivity image data and low sensitivity image data ofwhich exposure times are different, and the image processing unit mayfurther divide an illumination area of an image obtained from the lowsensitivity image data or the high sensitivity image data into 2N levelsof image areas ranging from an image area of which illumination is thelowest to an image area of which illumination is the highest, and maygenerate a combined image using N sets of high sensitivity image datarespectively corresponding to N levels of low illumination image areasin N levels of low illumination image areas ranging from the image areaof which illumination is the lowest to the image area of whichillumination is the N-th lowest, and using N sets of low sensitivityimage data respectively corresponding to N levels of high illuminationimage areas in the N levels of high illumination image areas rangingfrom the image area of which illumination is the (N+1)-th lowest to theimage area of which illumination is the highest. In this case, N is 2,and there can be such relationship that the image-capturing time forobtaining the high sensitivity image data and the low sensitivity imagedata of the first set is twice the image-capturing time for obtainingthe high sensitivity image data and the low sensitivity image data ofthe second set. It should be noted that such configuration can also beexpanded to a case where N is equal to or more than 3. Even with suchconfiguration, the dynamic range can be expanded. In addition, an imageof which dynamic range is expanded to 2N levels from N sets of imagedata can be obtained, and it is easy to cope with image-capturing of amotion picture. In the conventional technique, as described above, onlyan image of which dynamic range is expanded to N levels from N sets ofimage data can be obtained.

In the image-capturing device according to the first aspect or the thirdaspect of the present disclosure including the various kinds ofpreferable configurations explained above, a gap may not exist betweenthe first filter, the second filter, the third filter, and the fourthfilter. Such configuration can be easily achieved by using an exposuremask with a corner serif pattern when at least one of the filters of thefirst filter, the second filter, the third filter, and the fourth filteris formed using a photolithography technique.

Further, in the image-capturing device according to the first aspect orthe third aspect of the present disclosure including the various kindsof preferable configurations explained above, the fourth filter may beconfigured to have a three-layer stacked layer structure including afirst material layer made of a first material constituting a firstfilter, a second material layer made of a second material constituting asecond filter, and a third material layer made of a third materialconstituting a third filter. Alternatively, the fourth filter may beconfigured to be made of a material made by mixing the first materialconstituting the first filter, the second material constituting thesecond filter, and the third material constituting the third filter.Alternatively, the fourth filter may be configured to have a two-layerstacked layer structure including a first material layer made of a firstmaterial constituting a first filter and a third material layer made ofa third material constituting a third filter. Alternatively, the fourthfilter may be configured to be made of a material made by mixing thefirst material constituting the first filter and the third materialconstituting the third filter. In such case, the spectral characteristicof the fourth filter is, for example, spectral characteristic such asmagenta. Alternatively, the fourth filter may be a material constitutingan on-chip lens having almost uniform spectral transmission rate overthe visible light range, or a transparent material having a spectraltransmission rate characteristic suitable for spectral visualsensitivity characteristics of a human.

In the image-capturing device according to the second aspect of thepresent disclosure including the various kinds of preferableconfigurations explained above, the neutral density filter may beconfigured to have a spectral characteristic equal to the spectralcharacteristic of the filter made by stacking the first filter and thethird filter. In the image-capturing device according to the fourthaspect of the present disclosure, the neutral density filter may beconfigured to have a spectral characteristic equal to the spectralcharacteristic the neutral density filter may be configured to have athree-layer stacked layer structure including the first material layermade of the first material constituting the first filter, the secondmaterial layer made of the second material constituting the secondfilter, and the third material layer made of the third materialconstituting the third filter of the image-capturing device according tothe first aspect or the third aspect of the present disclosure.Alternatively, the neutral density filter may be configured to be madeof a material made by mixing the first material constituting the firstfilter, the second material constituting the second filter, and thethird material constituting the third filter of the image-capturingdevice according to the first aspect or the third aspect of the presentdisclosure. Alternatively, a material constituting an on-chip lenshaving almost uniform spectral transmission rate over the visible lightrange, or a transparent material suitable for spectral visualsensitivity characteristics of a human can be exemplified as a fourthfilter.

In the image-capturing device according to the first aspect or the thirdaspect of the present disclosure, the first wavelength band may be, forexample, a wavelength band such as 350 nm to 550 nm (mainly a wavelengthband for blue color), and the peak wavelength band of the firstwavelength band may be, for example, 430 nm to 480 nm. The secondwavelength band may be, for example, a wavelength band such as 450 nm to650 nm (mainly a wavelength band for green color), and the peakwavelength band of the second wavelength band may be, for example, 500nm to 550 nm. The third wavelength band may be, for example, awavelength band such as 550 nm to 750 nm (mainly a wavelength band forred color), and the peak wavelength band of the third wavelength bandmay be, for example, 580 nm to 620 nm. However, the embodiments are notlimited thereto. In some cases, the filter may be a color filter forpassing a particular wavelength such as cyan, magenta, and yellow.

In the image-capturing device according to the first aspect of thepresent disclosure, an arrangement of four image-capturing elements inone image-capturing element unit may be Bayer arrangement. Morespecifically, the image-capturing element units are arranged in atwo-dimensional matrix form, and in a case where the image-capturingelement units are arranged in a first direction and a second directionperpendicular to the first direction, for example, the image-capturingelements are arranged in any given image-capturing element unit asfollows: the third image-capturing element and the fourthimage-capturing element are arranged adjacently to each other along thefirst direction, the first image-capturing element and the secondimage-capturing element are arranged adjacently to each other along thefirst direction, the first image-capturing element and the fourthimage-capturing element are arranged adjacently to each other along thesecond direction, and the second image-capturing element and the thirdimage-capturing element are arranged adjacently to each other along thesecond direction. In the image-capturing device according to the thirdaspect of the present disclosure, an arrangement of four image-capturingelements in one image-capturing element sub-unit may be Bayerarrangement. More specifically, for example, in one image-capturingelement sub-unit, the third image-capturing element and the fourthimage-capturing element are arranged adjacently to each other along thefirst direction, the first image-capturing element and the secondimage-capturing element are arranged adjacently to each other along thefirst direction, the first image-capturing element and the fourthimage-capturing element are arranged adjacently to each other along thesecond direction, and the second image-capturing element and the thirdimage-capturing element are arranged adjacently to each other along thesecond direction. Further, in the image-capturing device according tothe second aspect of the present disclosure, an arrangement of fourimage-capturing elements in one image-capturing element unit may be anarrangement similar to the Bayer arrangement. More specifically, forexample, in any given image-capturing element unit, the thirdimage-capturing element and the fourth image-capturing element arearranged adjacently to each other along the first direction, the firstimage-capturing element and the second image-capturing element arearranged adjacently to each other along the first direction, the firstimage-capturing element and the fourth image-capturing element arearranged adjacently to each other along the second direction, and thesecond image-capturing element and the third image-capturing element arearranged adjacently to each other along the second direction.

In the image-capturing device according to the first aspect or the thirdaspect of the present disclosure, the fourth image-capturing element mayreceive light in the first wavelength band, the second wavelength band,and the third wavelength band. In other words, this can also be said tobe receiving light in the visible light range.

In the image-capturing device according to the first aspect or the thirdaspect of the present disclosure, the light transmission rate of thefourth filter may be less than the light transmission rate of the firstfilter, the light transmission rate of the second filter, and the lighttransmission rate of the third filter, and more specifically, where thelight transmission rate of the fourth filter is denoted as T₄, and anaverage value of the light transmission rate of the first filter, thelight transmission rate of the second filter, and the light transmissionrate of the third filter is denoted as T₁₂₃, for example, the lighttransmission rate of the fourth filter is configured to satisfy0.01≦T₄/T₁₂₃≦0.25.

In the image-capturing device according to the first aspect or the thirdaspect of the present disclosure, in a case where the difference betweenthe average luminance of the high sensitivity image data in the lowillumination area in the image-capturing scene and the average luminanceof the low sensitivity image data in the high illumination area in theimage-capturing scene is equal to or more than a predetermined value,the combined image may be generated using the high sensitivity imagedata corresponding to the low illumination image area in the lowillumination image area obtained from the low sensitivity image data orthe high sensitivity image data and using the low sensitivity image datacorresponding to the high illumination image area in the highillumination image area obtained from the low sensitivity image data orthe high sensitivity image data, and on the other hand, in a case wherethe difference between the average luminance of the high sensitivityimage data and the average luminance of the low sensitivity image datais less than a predetermined value, the image can also be obtained usingonly the high sensitivity image data corresponding to the lowillumination image area.

In the image-capturing device according to the first aspect of thepresent disclosure, the second image-capturing element may be assumed tobe located in an area where the fourth image-capturing element islocated, and an output that may be considered to be output by the secondimage-capturing element that is assumed as above (for the sake ofconvenience, this will be referred to as “assumed second image-capturingelement”) is interpolated from an output from a second image-capturingelement adjacent to the assumed second image-capturing element, so thatit is preferable to obtain an output equivalent to an output from animage-capturing element in an ordinary image-capturing device of theBayer arrangement (one red image-capturing element, two greenimage-capturing elements, and one blue image-capturing element. This isalso applicable to the following cases). Such interpolation processingwill be referred to as “interpolation processing of assumed secondimage-capturing element” for the sake of convenience. The fourthimage-capturing element may be assumed to be located in an area wherethe first image-capturing element is located, and an output that may beconsidered to be output by the fourth image-capturing element that isassumed as above (for the sake of convenience, this will be referred toas “assumed first (fourth) image-capturing element”) may be interpolatedfrom an output of a fourth image-capturing element adjacent to theassumed first (fourth) image-capturing element, the fourthimage-capturing element may be assumed to be located in an area wherethe second image-capturing element is located, and an output that may beconsidered to be output by the fourth image-capturing element that isassumed as above (for the sake of convenience, this will be referred toas “assumed second (fourth) image-capturing element”) may beinterpolated from an output of a fourth image-capturing element adjacentto the assumed second (fourth) image-capturing element, and the fourthimage-capturing element may be assumed to be located in an area wherethe third image-capturing element is located, and an output that may beconsidered to be output by the fourth image-capturing element that isassumed as above (for the sake of convenience, this will be referred toas “assumed third (fourth) image-capturing element”) may be interpolatedfrom an output of a fourth image-capturing element adjacent to theassumed third (fourth) image-capturing element, so that it is preferableto obtain an output based on the fourth image-capturing element similarto an output from an image-capturing element in an ordinaryimage-capturing device of the Bayer arrangement. With the aboveinterpolation processing, the data configuration and the data structureof the low sensitivity image data and the data configuration and thedata structure of the high sensitivity image data can be configured tobe the same while the original number of pixels is not changed, andtherefore, various kinds of signal processing can be simplified.

In the image-capturing device according to the second aspect of thepresent disclosure, the fourth image-capturing element may be assumed tobe located in an area where the first image-capturing element islocated, and an output that may be considered to be output by the fourthimage-capturing element that is assumed as above (for the sake ofconvenience, this will be referred to as “assumed first (fourth)image-capturing element”) may be interpolated from an output of a fourthimage-capturing element adjacent to the assumed first (fourth)image-capturing element, the fourth image-capturing element may beassumed to be located in an area where the second image-capturingelement is located, and an output that may be considered to be output bythe fourth image-capturing element that is assumed as above (for thesake of convenience, this will be referred to as “assumed second(fourth) image-capturing element”) may be interpolated from an output ofa fourth image-capturing element adjacent to the assumed second (fourth)image-capturing element, and the fourth image-capturing element may beassumed to be located in an area where the third image-capturing elementis located, and an output that may be considered to be output by thefourth image-capturing element that is assumed as above (for the sake ofconvenience, this will be referred to as “assumed third (fourth)image-capturing element”) may be interpolated from an output of a fourthimage-capturing element adjacent to the assumed third (fourth)image-capturing element, so that it is preferable to obtain an outputbased on the fourth image-capturing element similar to an output from animage-capturing element in an ordinary image-capturing device of theBayer arrangement. Likewise, the third image-capturing element may beassumed to be located in an area where the first image-capturing elementis located, and an output that may be considered to be output by thethird image-capturing element that is assumed as above (for the sake ofconvenience, this will be referred to as “assumed first (third)image-capturing element”) may be interpolated from an output of a thirdimage-capturing element adjacent to the assumed first (third)image-capturing element, the third image-capturing element may beassumed to be located in an area where the second image-capturingelement is located, and an output that may be considered to be output bythe third image-capturing element that is assumed as above (for the sakeof convenience, this will be referred to as “assumed second (third)image-capturing element”) may be interpolated from an output of a thirdimage-capturing element adjacent to the assumed second (third)image-capturing element, and the third image-capturing element may beassumed to be located in an area where the fourth image-capturingelement is located, and an output that may be considered to be output bythe third image-capturing element that is assumed as above (for the sakeof convenience, this will be referred to as “assumed fourth (third)image-capturing element”) may be interpolated from an output of a thirdimage-capturing element adjacent to the assumed fourth (third)image-capturing element, so that it is preferable to obtain an outputbased on the third image-capturing element similar to an output from animage-capturing element in an ordinary image-capturing device of theBayer arrangement. Likewise, the second image-capturing element may beassumed to be located in an area where the first image-capturing elementis located, and an output that may be considered to be output by thesecond image-capturing element that is assumed as above (for the sake ofconvenience, this will be referred to as “assumed first (second)image-capturing element”) may be interpolated from an output of a secondimage-capturing element adjacent to the assumed first (second)image-capturing element, the second image-capturing element may beassumed to be located in an area where the third image-capturing elementis located, and an output that may be considered to be output by thesecond image-capturing element that is assumed as above (for the sake ofconvenience, this will be referred to as “assumed third (second)image-capturing element”) may be interpolated from an output of a secondimage-capturing element adjacent to the assumed third (second)image-capturing element, and the second image-capturing element may beassumed to be located in an area where the fourth image-capturingelement is located, and an output that may be considered to be output bythe second image-capturing element that is assumed as above (for thesake of convenience, this will be referred to as “assumed fourth(second) image-capturing element”) may be interpolated from an output ofa second image-capturing element adjacent to the assumed fourth (second)image-capturing element, so that it is preferable to obtain an outputbased on the second image-capturing element similar to an output from animage-capturing element in an ordinary image-capturing device of theBayer arrangement. Likewise, the first image-capturing element may beassumed to be located in an area where the second image-capturingelement is located, and an output that may be considered to be output bythe first image-capturing element that is assumed as above (for the sakeof convenience, this will be referred to as “assumed second (first)image-capturing element”) may be interpolated from an output of a firstimage-capturing element adjacent to the assumed second (first)image-capturing element, the first image-capturing element may beassumed to be located in an area where the third image-capturing elementis located, and an output that may be considered to be output by thefirst image-capturing element that is assumed as above (for the sake ofconvenience, this will be referred to as “assumed third (first)image-capturing element”) may be interpolated from an output of a firstimage-capturing element adjacent to the assumed third (first)image-capturing element, and the first image-capturing element may beassumed to be located in an area where the fourth image-capturingelement is located, and an output that may be considered to be output bythe first image-capturing element that is assumed as above (for the sakeof convenience, this will be referred to as “assumed fourth (first)image-capturing element”) may be interpolated from an output of a firstimage-capturing element adjacent to the assumed fourth (first)image-capturing element, so that it is preferable to obtain an outputbased on the first image-capturing element similar to an output from animage-capturing element in an ordinary image-capturing device of theBayer arrangement. With the above interpolation processing, the dataconfiguration and the data structure of the low sensitivity image dataand the data configuration and the data structure of the highsensitivity image data can be configured to be the same while theoriginal number of pixels is not changed, and therefore, various kindsof signal processing can be simplified.

In the image-capturing device according to the third aspect of thepresent disclosure, where a total output of a cluster including fourfirst image-capturing elements of four image-capturing element sub-unitsis referred to as “first image-capturing element cluster output,” atotal output of a cluster including four second image-capturing elementsthereof is referred to as “second image-capturing element clusteroutput,” and a total output of a cluster including four thirdimage-capturing elements thereof is referred to as “thirdimage-capturing element cluster output,” “the interpolation processingof the assumed second image-capturing element” mentioned above may beperformed on the basis of the first image-capturing element clusteroutput, the second image-capturing element cluster output, and the thirdimage-capturing element cluster output. Accordingly, an outputequivalent to the output from image-capturing elements in an ordinaryimage-capturing device of the Bayer arrangement (one red image-capturingelement, two green image-capturing elements, and one blue colorimage-capturing element) can be obtained from the first image-capturingelement cluster output, the second image-capturing element clusteroutput, and the third image-capturing element cluster output. An outputequivalent to outputs from four image-capturing elements in an ordinaryimage-capturing device of the Bayer arrangement can be obtained withoutperforming adding processing on the basis of the fourth image-capturingelement constituting the first image-capturing element sub-unit, thefourth image-capturing element constituting the second image-capturingelement sub-unit, the fourth image-capturing element constituting thethird image-capturing element sub-unit, and the fourth image-capturingelement constituting the fourth image-capturing element sub-unit.Accordingly, the data configuration and the data structure of the lowsensitivity image data obtained from ¼ of the original number of pixelscan be the same as the data configuration and the data structure of thehigh sensitivity image data, and various kinds of signal processing canbe simplified. The high sensitivity image data are subjected to thefour-pixel adding processing, but the low sensitivity image data are notsubjected to the adding processing. Therefore, there is an advantage inthat the sensitivity difference can be increased to four times.

In the image-capturing device according to the fourth aspect of thepresent disclosure, the second image-capturing element may be assumed tobe located in an area where the first image-capturing element islocated, and an output that may be considered to be output by the secondimage-capturing element that is assumed as above (for the sake ofconvenience, this will be referred to as “assumed first image-capturingelement”) may be interpolated from an output of a second image-capturingelement adjacent to the assumed first image-capturing element, and thefirst image-capturing element may be assumed to be located in an areawhere the second image-capturing element is located, and an output thatmay be considered to be output by the first image-capturing element thatis assumed as above (for the sake of convenience, this will be referredto as “assumed first image-capturing element”) may be interpolated froman output of a second image-capturing element adjacent to the assumedfirst image-capturing element, so that low sensitivity image data andhigh sensitivity image data can be obtained.

In the image-capturing device according to the first aspect to thefourth aspect of the present disclosure, a low illumination image areaand a high illumination image area are obtained from the low sensitivityimage data or the high sensitivity image data, and in a case where theluminance value of the high sensitivity image data in any givenillumination area is saturated, or close to saturation, this area isdetermined to be a high illumination area, and the luminance value ofthe low sensitivity image data is used. The method for deriving the lowillumination image area and the high illumination image area may be awell-known method.

With the image-capturing device according to the first aspect or thethird aspect of the present disclosure, a single-color (gray) image canalso be obtained. In this case, the first gain adjustment unit mayperform gain adjustment on the outputs from the first image-capturingelement, the second image-capturing element, and the thirdimage-capturing element, and the spectral characteristic based on theoutputs preferably match the spectral characteristic of the fourthfilter provided in the fourth image-capturing element.

Examples of photoelectric conversion elements include a CCD element, aCMOS image sensor, a CIS (Contact Image Sensor), a CMD (ChargeModulation Device)-type signal amplification image sensor. Examples ofimage-capturing elements include a front-illuminated image-capturingelement and a back-illuminated image-capturing element. Further, theimage-capturing device according to the present disclosure mayconstitute, for example, a digital still camera, a video camera, acamcorder, a so-called camera-equipped cellular phone. Theimage-capturing device may have a lens system. The lens system may be asingle-focus lens or a so-called zoom lens. The configuration and thestructure of the lens and the lens system may be determined on the basisof the specification required for the lens system.

First Embodiment

The first embodiment relates to the image-capturing device according tothe first aspect of the present disclosure. A conceptual diagram of theimage-capturing unit of the image-capturing device of the firstembodiment is shown in FIG. 1A. A conceptual diagram for explainingoperation of the image-capturing unit is shown in FIG. 2A, FIG. 2B, FIG.3A, FIG. 3B, FIG. 4 and FIG. 5. A conceptual diagram of theimage-capturing device is shown in FIG. 7A. A schematic partial crosssectional view of an image-capturing element is shown in FIG. 7B. lightreception characteristics of the first image-capturing element, thesecond image-capturing element, the third image-capturing element, andthe fourth image-capturing element are schematically shown in FIG. 1B.Further, the image-capturing state of the image-capturing device of thefirst embodiment is schematically shown in FIG. 6.

The image-capturing device 1 of the first embodiment is animage-capturing device for obtaining a combined image in which a portionof the image is a single-color image and the remaining portion thereofis a color image. In FIG. 7B, the image-capturing device 1 of the firstembodiment includes an image-capturing unit 30 and an image processingunit 11, wherein the image-capturing unit 30 includes image-capturingelement units 31 arranged in a two-dimensional matrix form, and each ofthe image-capturing element units 31 includes: a first image-capturingelement 41 including a first filter 43 and a first photoelectricconversion element 42, and configured to receive light in a firstwavelength band, a second image-capturing element 51 including a secondfilter 53 and a second photoelectric conversion element 52, andconfigured to receive light in a second wavelength band having a peakwavelength longer than a peak wavelength of the first wavelength band, athird image-capturing element 61 including a third filter 63 and a thirdphotoelectric conversion element 62, and configured to receive light ina third wavelength band having a peak wavelength longer than a peakwavelength of the second wavelength band, and a fourth image-capturingelement 71 including a fourth filter 73 and a fourth photoelectricconversion element 72, and configured to receive light in the firstwavelength band, the second wavelength band, and the third wavelengthband. It should be noted that the number of pixels of theimage-capturing unit 30 is, for example, VGA (640×480), and theimage-capturing unit 30 captures a motion picture. This is applicable toimage-capturing devices in the following embodiments.

In this case, although not limited to the following configuration, thefirst wavelength band is 350 nm to 550 nm (mainly blue color wavelengthband), and the peak wavelength of the first wavelength band is 450 nm,and the first image-capturing element 41 is a blue color image-capturingelement; the second wavelength band is 450 nm to 650 nm (mainly greencolor wavelength band), and the peak wavelength of the second wavelengthband is 530 nm, and the second image-capturing element 51 is a greencolor image-capturing element. Further, the third wavelength band is 550nm to 750 nm (mainly red color wavelength band), and the peak wavelengthof the third wavelength band is 600 nm, and the third image-capturingelement 61 is a red color image-capturing element.

The light transmission rate of the fourth filter 73 that functions as anND filter (Neutral Density Filter) is less than the light transmissionrate of the first filter 43, the light transmission rate of the secondfilter 53, and the light transmission rate of the third filter 63.Alternatively, the sensitivity of the fourth image-capturing element 71is less than the sensitivities of the first image-capturing element 41,the second image-capturing element 51, and the third image-capturingelement 61. FIG. 1B schematically illustrates the light receptioncharacteristics of the first image-capturing element, the secondimage-capturing element, the third image-capturing element, and thefourth image-capturing element. More specifically, where the lighttransmission rate of the fourth filter 73 is denoted as T₄, and theaverage value of the light transmission rate of the first filter 43, thelight transmission rate of the second filter 53, and the lighttransmission rate of the third filter 63 is denoted as T₁₂₃,T₄/T₁₂₃≈0.05 holds, although the light reception characteristics are notlimited thereto. The fourth filter 73 is made of a material obtained bymixing a pigment-based first material constituting the first filter 43,a pigment-based second material constituting the second filter 53, and apigment-based third material constituting the third filter 63, or thefourth filter 73 is made of a material obtained by mixing apigment-based first material constituting the first filter 43 and apigment-based third material constituting the third filter 63. Thethickness of the fourth filter 73 is about 1 μm, and the fourth filter73 can be formed on the basis of ordinary lithography technique byultraviolet irradiation. The fourth image-capturing element 71 is a grayimage-capturing element, and the output from the fourth image-capturingelement 71 is gray gradation information without any color information.

For example, the image-capturing elements 41, 51, 61, 71 havephotoelectric conversion elements 42, 52, 62, 72 provided on a siliconsemiconductor substrate 80 as shown in the partial cross sectional viewof FIG. 7B, and further, a first planarization film 82, filters 43, 53,63, 73, an on-chip lens 84, and a second planarization film 85 arestacked thereon. It should be noted that reference numeral 86 denotes alight shield layer, and reference numeral 88 denotes a wiring layer.

The photoelectric conversion elements 42, 52, 62, 72 are made of, forexample, CMOS image sensors. Further, for example, a video camera andthe like are constituted by the image-capturing device of the firstembodiment. As shown in the conceptual diagram of FIG. 7A, theimage-capturing device 1 has a lens system 20. In the lens system 20, animage-capturing lens 21, a diaphragm unit 22, and an imaging lens 23 areaccommodated in the lens barrel, and function as the zoom lens. Theimage-capturing lens 21 is a lens for condensing incidence light from asubject. The image-capturing lens 21 includes, for example, a focus lensfor focus and a zoom lens for enlarging a subject, and in general, theimage-capturing lens 21 is achieved by a combination of multiple lensesfor correcting chromatic aberration and the like. The diaphragm unit 22has a narrowing down function for adjusting the quantity of lightcondensed, and in general, the diaphragm unit 22 is constituted by acombination of multiple diaphragm blades in a plate-like shape. At leastat the position of the diaphragm unit 22, the light from any given pointof the subject becomes parallel light. The imaging lens 23 focuses thelight onto the image-capturing unit 30. The image-capturing unit 30 isprovided inside of a camera main body 2. The camera main body 2 includesnot only the image-capturing unit 30 but also, for example, an imageprocessing unit 11 and an image storage unit 12. Then, image data areformed on the basis of an output (electric signal) from theimage-capturing unit 30. The image processing unit 11 constituted by awell-known circuit converts an electric signal which is output from theimage-capturing unit 30 into image data, and records the image data to awell-known image storage unit 12. It should be noted that a band passfilter (not shown) for shielding light other than visible light isprovided above the image-capturing element, or a seal glass surface ofan image-capturing element package, or in a lens barrel.

As shown in FIG. 1A, in the image-capturing device 1 of the firstembodiment, the arrangement of four image-capturing elements 41, 51, 61,71 (indicated by squares of solid lines and dotted lines in FIG. 1A) ofany given image-capturing element unit 31 (indicated by square of solidline in FIG. 1A) is Bayer arrangement. More specifically, theimage-capturing element units 31 are arranged in a two-dimensionalmatrix form, but the image-capturing element units 31 are arranged inthe first direction and the second direction perpendicular to the firstdirection, and in any given image-capturing element unit 31, theimage-capturing elements 41, 51, 61, 71 are arranged as follows: thethird image-capturing element 61 and the fourth image-capturing element71 are arranged adjacently to each other along the first direction, thefirst image-capturing element 41 and the second image-capturing element51 are arranged adjacently to each other in the first direction, thefirst image-capturing element 41 and the fourth image-capturing element71 are arranged adjacently to each other in the second direction, andthe second image-capturing element 51 and the third image-capturingelement 61 are arranged adjacently to each other in the seconddirection.

Then, the image processing unit 11 generates high sensitivity image datafrom outputs of the first image-capturing element 41, the secondimage-capturing element 51, and the third image-capturing element 61,and generates low sensitivity image data from an output from the fourthimage-capturing element 71.

More specifically, when a certain image-capturing scene is captured, thefirst image-capturing element 41, the second image-capturing element 51,and the third image-capturing element 61 output signal values “B”, “G₁”,“R,” respectively, as shown in FIG. 2A. On the other hand, the fourthimage-capturing element 71 outputs the signal value “Gy₄” as shown inFIG. 2B.

The light reception sensitivities of the first photoelectric conversionelement 42 constituting the first image-capturing element 41, the secondphotoelectric conversion element 52 constituting the secondimage-capturing element 51, and the third photoelectric conversionelement 62 constituting the third image-capturing element 61 are equally“1” for the sake of convenience and for simplifying the explanation. Thelight reception sensitivity of the fourth photoelectric conversionelement 72 constituting the fourth image-capturing element 71 is “⅛” forthe sake of convenience. This is considered to be also applicable to thefollowing explanation. More specifically, the ratio between the averageaccumulation electrical-charge quantity of the first image-capturingelement 41, the second image-capturing element 51, and the thirdimage-capturing element 61 and the accumulation electrical-chargequantity of the fourth image-capturing element 71 is 8:1 (18 dB). Due tothe sensitivity difference of 18 dB, low sensitivity image data forthree levels of exposure are obtained. This is basically also applicableto the following explanation.

Each image-capturing element having received the light in the state asshown in FIGS. 2A and 2B transmits an output (electric signal) based onthe received light quantity to the image processing unit 11. In theimage processing unit 11, for example, the interpolation processing isperformed on the basis of the following method.

More specifically, the second image-capturing element 51 is assumed tobe located in an area where the fourth image-capturing element 71 islocated, and an output that may be considered to be output from thesecond image-capturing element 51 that has been assumed (assumed secondimage-capturing element 51) is interpolated from an output from a secondimage-capturing element 51 adjacent to the assumed secondimage-capturing element 51. For example, in FIG. 2A, an output that maybe considered to be output from the (i+1, j+1)-th assumed secondimage-capturing element 51 is derived as an average value “G′₂” ofoutputs from four image-capturing elements, that is, a (i+1, j)-thsecond image-capturing element 51, a (i+2, j)-th second image-capturingelement 51, a (i+1, j+1)-th second image-capturing element 51, and a(i+2, j+1)-th second image-capturing element 51. A reference symbolhaving an apostrophe “'” attached thereto basically indicates an outputobtained by interpolation. In this manner, outputs (R, G₁, G′₂, B)equivalent to the outputs from the image-capturing elements in anordinary image-capturing device of the Bayer arrangement (one red colorimage-capturing element, two green color image-capturing elements, andone blue color image-capturing element) can be obtained (see FIG. 3A),and the high sensitivity image data can be obtained from the entireobtained outputs.

On the other hand, the fourth image-capturing element 71 is assumed tobe located in an area where the first image-capturing element 41 islocated, and an output “Gy₁” that may be considered to be output fromthe fourth image-capturing element 71 that has been assumed (assumedfirst (fourth) image-capturing element) is interpolated from an outputfrom a fourth image-capturing element 71 adjacent to the assumed first(fourth) image-capturing element. More specifically, for example, theoutput value “Gy′₁” from the (i+1, j+1)-th assumed first (fourth)image-capturing element is derived from an average value of the outputsfrom the (i+1, j+1)-th fourth image-capturing element 71 and the (i+1,j+2)-th fourth image-capturing element 71. The fourth image-capturingelement 71 is assumed to be located in an area where the secondimage-capturing element 51 is located, and an output that may beconsidered to be output from the fourth image-capturing element 71 thathas been assumed (assumed second (fourth) image-capturing element) isinterpolated from an output from a fourth image-capturing element 71adjacent to the assumed second (fourth) image-capturing element. Morespecifically, the output value “Gy′₂” of the (i+1, j+1)-th assumedsecond (fourth) image-capturing element is derived from an average valueof outputs from the (i, j+1)-th fourth image-capturing element 71, the(i+1, j+1)-th fourth image-capturing element 71, the (i, j+2)-th fourthimage-capturing element 71, and, the (i+1, j+2)-th fourthimage-capturing element 71. Further, the fourth image-capturing element71 is assumed to be located in an area where the third image-capturingelement 61 is located, and an output that may be considered to be outputfrom the fourth image-capturing element 71 that has been assumed(assumed third (fourth) image-capturing element) is interpolated from anoutput from a fourth image-capturing element 71 adjacent to the assumedthird (fourth) image-capturing element. More specifically, the outputvalue “Gy′₃” of the (i+1, j+1)-th assumed third (fourth) image-capturingelement is derived from an average value of outputs from the (i, j+1)-thfourth image-capturing element 71 and the (i+1, j+1)-th fourthimage-capturing element 71. In this manner, outputs (Gy′₁, Gy′₂, Gy′₃,Gy₄) based on the fourth image-capturing element 71 similar to theoutputs from the image-capturing elements in an ordinary image-capturingdevice of the Bayer arrangement can be obtained (see FIG. 3B), and thelow sensitivity image data can be obtained from the entire obtainedoutputs.

It is to be understood that the interpolation method (bi-linear method)explained above is merely an example, and can be changed as necessary. Amethod adopting an output given by an adjacent second image-capturingelement 51 as an output of the assumed second image-capturing element 51as it is (nearest neighbor method), and further, a more sophisticatedpixel interpolation algorithm using direction selection algorithm may beapplied. The algorithm may be determined by the target frame rate andthe image processing cost of the entire system. This is also applicableto the following explanation.

With such processing, the data configuration and the data structure ofthe low sensitivity image data and the data configuration and the datastructure of the high sensitivity image data can be configured to be thesame while the original number of pixels is not changed, and therefore,various kinds of signal processing can be simplified, and an image canbe generated that is combined by HDR imaging with a processing engine atan extremely low cost. Since the simplest periodicity and symmetricproperty are maintained as image processing, image quality degradationsuch as false color and jaggy is less likely to occur when a combinedimage is formed.

Subsequently, the image processing unit 11 generates a combined imageusing high sensitivity image data corresponding to a low illuminationimage area in the low illumination image area obtained from the lowsensitivity image data or the high sensitivity image data, and using lowsensitivity image data corresponding to a high illumination image areain the high illumination image area obtained from the low sensitivityimage data or the high sensitivity image data. More specifically, forexample, the low illumination image area and the high illumination imagearea are extracted from the high sensitivity image data. Morespecifically, in a case where the luminance average value of a pixel ofhigh sensitivity image data in any given size of area is equal to ormore than a value defined in advance (for example, in a case where theluminance value after the AD conversion is saturated, or in a case wherethe luminance value is 90% or higher with respect to the saturationlevel, etc.), this area may be adopted as the high illumination imagearea, and in a case where the illumination average value is less thanthe value defined in advance, this area may be adopted as the lowillumination image area. For example, in FIG. 4, an area encircled bydouble lines with shade is considered to be the high illumination imagearea.

Then, the combined image is generated using the high sensitivity imagedata corresponding to the low illumination image area in the lowillumination image area and using the low sensitivity image datacorresponding to the high illumination image area in the highillumination image area. This state is schematically shown in FIG. 5.

As schematically shown in FIG. 6, an image-capturing scene is consideredin which, an apple and a banana is placed on a table arranged in a darkroom, and there are trees outdoors outside of windows where it isbright. In such image-capturing scene, when an image-capturing condition(exposure and shutter speed) is adjusted for a dark room, theimage-capturing state in the room is appropriate, but “blown outhighlights” may occur in the scenery outdoors outside of the windows(see a drawing at the upper side at the left hand side of FIG. 6). Onthe other hand, when the image-capturing condition (exposure and gain)is adjusted for the outdoors where it is bright, the image-capturingstate inside of the room becomes “blocked up shadows” (see a drawing atthe lower side at the left hand side of FIG. 6).

In the image-capturing device of the first embodiment, for example, theshutter speed is adjusted so that the maximum luminance value in thehigh illumination area of the low sensitivity image data is about 80%output of the AD conversion full range. It should be noted that theframe rate is fixed, which is, for example, 30 fps. With suchprocessing, the output from the fourth image-capturing element 71 is notsaturated in the high illumination image area, and although it is grayscale, a luminance distribution with a preferable linearity can beobtained. In the high illumination image area, at least one of the firstimage-capturing element 41, the second image-capturing element 51, andthe third image-capturing element 61 is saturated, and a part or all ofthe color information is lost. On the other hand, in the lowillumination image area, the first image-capturing element 41, thesecond image-capturing element 51, and the third image-capturing element61 are not saturated, and therefore, a luminance distribution and colorinformation can be obtained with a preferable linearity.

Both of the low sensitivity image data and the high sensitivity imagedata are obtained by one image-capturing process, and a combined imageis generated using the high sensitivity image data corresponding to thedark room (low illumination image area) for the dark room (lowillumination image area) and using the low sensitivity image datacorresponding to the bright outdoors (high illumination image area) forthe bright outdoors (high illumination image area). This state isschematically shown in the drawing at the right hand side of FIG. 6. Inaccordance with the dynamic range (for example, RGB 8 bit) of a displayapparatus for displaying a combined image or a printer for printing acombined image, so-called tone mapping processing is performed toconvert the luminance range in an image into the range suitable fordisplay, whereby an image combined by HDR imaging can be obtained. Theimage of bright outdoors (high illumination image area) is a singlecolor (black and white) image, and the image of the dark room (lowillumination image area) is a color image. Such combined image (imagecombined by HDR imaging) is required to have color on the entire screenof devices requiring high degree of quality such as ordinary digitalcameras and camcorders, and therefore, the image-capturing device of thefirst embodiment is difficult to be applied to such devices. However, inthe usage field of image-capturing devices such as a monitoring cameraand an on-board camera which are strongly required “not to lose anyinformation,” this would not cause any problem in practical point ofview. In addition, it is easy to support motion pictures. As describedabove, although the image-capturing device has, for example, the simpleconfiguration and structure equivalent to those of the image-capturingdevice having a conventional Bayer arrangement, the image-capturingdevice can easily obtain an image based on a high dynamic range imaging,and this means that an image by high dynamic range imaging can beobtained by using inexpensive image-capturing device.

It should be noted that an output (electrical-charge) generated by eachimage-capturing element in accordance with the received light quantityis sent to a column signal processing circuit. For example, the columnsignal processing circuit is provided for each column of theimage-capturing elements, and performs, for each column, signalprocessing such as noise reduction on the output that is output from onerow of image-capturing elements. More specifically, the column signalprocessing circuit performs signal processing such as correlated doublesampling (CDS) for eliminating fixed pattern noise unique to theimage-capturing element, signal amplification, and analog/digitalconversion. The horizontal driving circuit is constituted by, forexample, shift registers, and selects the column signal processingcircuits in order by successively outputting a horizontal scanningpulse, thus causing each of the column signal processing circuits tooutput a signal. The system control circuit receives data for, forexample, commanding an input clock and an operation mode, and outputsdata such as internal information about the image-capturing device. Morespecifically, the system control circuit generates a clock signal and acontrol signal serving as a basis of operation of the vertical drivecircuit, the column signal processing circuit, the horizontal drivingcircuit, and the like on the basis of the vertical synchronizationsignal, the horizontal synchronization signal, and master clock. Then,the system control circuit outputs these signals to the vertical drivecircuit, the column signal processing circuit, the horizontal drivingcircuit, and the like. The drive circuit for driving eachimage-capturing element is constituted by the peripheral circuits(vertical drive circuit, column signal processing circuit, horizontaldriving circuit and system control circuit) and the circuit provided foreach of the image-capturing elements.

Alternatively, the image-capturing element is driven on the basis of thecircuit of the first embodiment disclosed in JP 2005-323331 A, forexample.

Second Embodiment

By increasing the film thickness of the fourth filter 73, the opticaldensity (OD) can be increased. However, the film thicknesses of thefilters 43, 53, 63 of the image-capturing elements 41, 51, 61 aretypically thinner than 1 μm. Therefore, the film thickness of the fourthfilter 73 of the fourth image-capturing element 71 formed in the sameplane is also desirably about the same in view of the light condensingperformance of the image-capturing element. In view of the filterforming process, it is sometimes difficult to form a filter of whichfilm thickness is thick with a variation equal to or less than a certainlevel. The light transmission rate of the fourth filter 73 of which filmthickness is 1 μm according to the first embodiment is limited to about5%, and the outputs of the fourth image-capturing element 71 is about ⅛of the output of the first image-capturing element 41, the secondimage-capturing element 51, and the third image-capturing element 61,and this determines the dynamic range of the entire image-capturingdevice. Therefore, in reality, in order to support an image-capturingscene where there is still more great difference in the brightness, thefourth image-capturing element 71 may be desirable to attain lowersensitivity in some cases.

The second embodiment is a modification of the first embodiment. In theimage-capturing device of the second embodiment, a schematic partial topview of the light shield layer 86 when the image-capturing element isconsidered to be cut along arrow B-B of FIG. 7B is shown in FIG. 8. Asshown in FIG. 8, a first aperture portion 47 is formed between a firstfilter 43 and a first photoelectric conversion element 42, a secondaperture portion 57 is formed between a second filter 53 and a secondphotoelectric conversion element 52, a third aperture portion 67 isformed between a third filter 63 and a third photoelectric conversionelement 62, and a fourth aperture portion 77 is formed between a fourthfilter 73 and a fourth photoelectric conversion element 72, wherein thefourth aperture portion 77 is smaller than the first aperture portion47, the second aperture portion 57, and the third aperture portion 67.

For example, when the sizes of the first aperture portion 47, the secondaperture portion 57, the third aperture portion 67, and the fourthaperture portion 77 are determined so that the light quantityattenuation for the fourth image-capturing element 71 is twice the lightquantity attenuation for the first image-capturing element 41, thesecond image-capturing element 51, and the third image-capturing element61. The sensitivity of the fourth image-capturing element 71 is ⅛ (thefirst embodiment) to 1/16 (24 dB) of the sensitivities of the firstimage-capturing element 41, the second image-capturing element 51, andthe third image-capturing element 61. As described above, the quantityof light received by the fourth image-capturing element 71 can be lessthan the quantity of light received by the first image-capturing element41, the second image-capturing element 51, and the third image-capturingelement 61, and as a result, the dynamic range can be expanded withoutthickening the film thickness of the fourth filter 73.

Third Embodiment

The third embodiment is also a modification of the first embodiment. Inthe first embodiment, the sensitivity ratio of the first image-capturingelement 41, the second image-capturing element 51, and the thirdimage-capturing element 61, and the fourth image-capturing element 71 is1:⅛ (18 dB), but when the sensitivity ratio is fixed, it may bedifficult to cope with various kinds of image-capturing scenes, and thedynamic rang such as 18 dB may be in sufficient.

In the third embodiment, the output of the final stage is variablycontrolled by combining the gains that can be independently set betweenthe first image-capturing element 41, the second image-capturing element51, and the third image-capturing element 61, and the fourthimage-capturing element 71, whereby the dynamic range of the imagecombined by HDR imaging can be expanded, and the variable control of thedynamic range is enabled, whereby image-capturing scenes where there arevarious differences in the brightness can be supported. In animage-capturing scene, the difference in the brightness needs to beequal to or more than 32 dB in many cases.

More specifically, in the third embodiment, the image processing unit 11may include a first gain adjustment unit for adjusting outputs from thefirst image-capturing element 41, the second image-capturing element 51,and the third image-capturing element 61 and a second gain adjustmentunit for adjusting an output from the fourth image-capturing element 71,and in this case, where an adjustment coefficient for outputs from thefirst image-capturing element 41, the second image-capturing element 51,and the third image-capturing element 61 by the first gain adjustmentunit is denoted as Gn₁, and an adjustment coefficient for an output fromthe fourth image-capturing element 71 by the second gain adjustment unitis denoted as Gn₂, Gn₁/Gn₂=4 may hold although the present embodiment isnot limited thereto. More specifically, the range of the gain differencebetween the first image-capturing element 41, the second image-capturingelement 51, and the third image-capturing element 61, and the fourthimage-capturing element 71 is up to 12 dB.

FIG. 9A illustrates a sensitivity ratio of the first image-capturingelement 41, the second image-capturing element 51, the thirdimage-capturing element 61, and the fourth image-capturing element 71.FIG. 9B is a gain ratio of the first image-capturing element 41, thesecond image-capturing element 51, the third image-capturing element 61,and the fourth image-capturing element 71. FIG. 10A illustrates a finaloutput ratio of high sensitivity image data (a ratio of output to theoutside, which is also applicable to the following cases). FIG. 10Billustrates a final output ratio of low sensitivity image data. In FIG.9A or FIG. 23A explained later, numerals in the parentheses “( )” denotea sensitivity ratio. In FIG. 9B or FIG. 23B explained later, numerals inthe angle brackets “< >” denote a gain value. In FIGS. 10A, 10B or FIGS.24A, 28B, and 29 explained later, numerals in the square brackets “[ ]”denote a final output ratio. The sensitivity ratio of the firstimage-capturing element 41, the second image-capturing element 51, andthe third image-capturing element 61, and the fourth image-capturingelement 71 is 1:(⅛), and the gain ratio of the first image-capturingelement 41, the second image-capturing element 51, and the thirdimage-capturing element 61, and the fourth image-capturing element 71 is4:1. Therefore, the final output ratio of the first image-capturingelement 41, the second image-capturing element 51, and the thirdimage-capturing element 61, and the fourth image-capturing element 71 is1:( 1/32), which is 30 dB.

The values of the adjustment coefficients Gn₁, Gn₂ may be fixed values,or may be automatically or manually changeable in accordance with theimage-capturing scene and the difference between the average luminancein the high sensitivity image data and the average luminance in the lowsensitivity image data, and therefore, image-capturing can be done withthe optimum dynamic range in accordance with various kinds of bright anddark scenes.

It should be noted that the second embodiment and the third embodimentmay be combined.

Fourth Embodiment

The fourth embodiment also relates to a modification of the firstembodiment. In a recent image-capturing device, images can be capturedwith a frame rate equal to or more than twice an ordinary frame rate. Inthe fourth embodiment, an image-capturing device capable of capturing animage with such high frame rate (HFR), that is, an image-capturingdevice capable of capturing an image with HFR which is twice the framerate (60 fps), is used.

More specifically, in the image-capturing device of the fourthembodiment, image processing unit 11 generates N sets of highsensitivity image data and low sensitivity image data of which exposuretimes are different, and the image processing unit 11 further divides anillumination area of an image obtained from the low sensitivity imagedata or the high sensitivity image data into 2N levels of areas rangingfrom an area of which illumination is the lowest to an area of whichillumination is the highest, and generates a combined image using N setsof high sensitivity image data respectively corresponding to N levels oflow illumination image areas in N levels of low illumination image areasranging from the area of which illumination is the lowest to the area ofwhich illumination is the N-th lowest, and using N sets of lowsensitivity image data respectively corresponding to N levels of highillumination image areas in N levels of high illumination image areasranging from the area of which illumination is the (N+1)-th lowest tothe area of which illumination is the highest. In the fourth embodiment,N is 2, and there is such relationship that the image-capturing time forobtaining the high sensitivity image data and the low sensitivity imagedata of the first set is twice the image-capturing time for obtainingthe high sensitivity image data and the low sensitivity image data ofthe second set.

FIG. 11A schematically illustrates an image-capturing state (exposuretime) in accordance with elapse of time. FIG. 11B schematicallyillustrates outputs from the first image-capturing element 41, thesecond image-capturing element 51, and the third image-capturing element61, and the fourth image-capturing element 71. In the fourth embodiment,in the first image-capturing process, the image processing unit 11generates one set of high sensitivity image data (indicated as “highsensitivity 1” in FIG. 11B) and low sensitivity image data (indicated as“low sensitivity 1” in FIG. 11B), and in the second image-capturingprocess, the image processing unit 11 generates another set of highsensitivity image data (indicated as “high sensitivity 2” in FIG. 11B)and low sensitivity image data (indicated as “low sensitivity 2” in FIG.11B). When the outputs from the first image-capturing element 41, thesecond image-capturing element 51, and the third image-capturing element61 are “1” in the first image-capturing process, the outputs from thefirst image-capturing element 41, the second image-capturing element 51,and the third image-capturing element 61 are “½” in the secondimage-capturing process. The output from the fourth image-capturingelement 71 in the first image-capturing process is “⅛,” and the outputfrom the fourth image-capturing element 71 in the second image-capturingprocess is “ 1/16.” Then, on the basis of the outputs from theseimage-capturing elements, two sets of high sensitivity image data andlow sensitivity image data are generated.

More specifically, the image processing unit 11 divides an illuminationarea of an image obtained from the low sensitivity image data or thehigh sensitivity image data into 4 levels of areas ranging from an areaof which illumination is the lowest to an area of which illumination isthe highest, and generates a combined image using two sets of highsensitivity image data respectively corresponding to two levels of lowillumination image areas, that is, high sensitivity image data obtainedfrom the “high sensitivity 1” and the “high sensitivity 2,” in twolevels of low illumination image areas ranging from the area of whichillumination is the lowest to the area of which illumination is thesecond lowest, and using two sets of low sensitivity image datarespectively corresponding to two levels of high illumination imageareas, that is, low sensitivity image data obtained from the “lowsensitivity 1” and the “low sensitivity 2,” in two levels of highillumination image areas ranging from the area of which illumination isthe third lowest to the fourth area of which illumination is thehighest.

As a result, the dynamic range of the image combined by HDR imaging is24 dB. It should be noted that, in the example as shown in FIG. 11B, thehigh sensitivity image data in the first image-capturing process aresaturated, on the other hand, the high sensitivity image data in thesecond image-capturing process are not saturated.

According to the fourth embodiment, the range in which the colorinformation can be obtained is twice as large as the conventional range,and the lower limit of detection of the gray image is also extended tothe half, and therefore, information can be obtained in a lowerillumination spatial area. Therefore, as compared to the image combinedby HDR imaging from an ordinary unit frame, not only the dynamic rangeis expanded, but also information about intermediate illumination isincreased, and therefore, the noise in the high dynamic range combinedimage is reduced, and an image made by highly precise HDR imaging can beformed from two unit frames.

It should be noted that the configuration and the structure of theimage-capturing device of the second embodiment can be applied to theimage-capturing device of the fourth embodiment, or the configurationand the structure of the image-capturing device of the third embodimentcan be applied to the image-capturing device of the fourth embodiment,or the configurations and the structures of the image-capturing devicesof the second embodiment and the third embodiment can be applied to theimage-capturing device of the fourth embodiment.

Fifth Embodiment

In a case where the first filter, the second filter, the third filterand the fourth filter are independently formed, more specifically, in acase where the first filter, the second filter, the third filter, andthe fourth filter of independent dot patterns are formed, a pattern of acorner portion of a resist material is withdrawn in an exposure maskhaving an ordinary rectangular opening when the resist material isexposed, and for this reason, in the order of exposure wavelength, thefour corners of the filter become round although a filter in arectangular shape is originally expected to be formed. As a result, agap is formed between a corner portion of the filter and a cornerportion of the filter, and there occurs a problem in that the colors maymix between image-capturing elements, and the overall reliability of theimage-capturing element is reduced.

The fifth embodiment relates to a modification of the first embodimentto the fourth embodiment. In the image-capturing device of the fifthembodiment, there is no gap between the first filter 43, the secondfilter 53, the third filter 63, and the fourth filter 73. When any oneof the first filter 43, the second filter 53, the third filter 63, andthe fourth filter 73 is formed using the photolithography technique,this kind of configuration can be easily achieved by using an exposuremask with corner serif patterns of which schematic partial top view isshown in FIG. 12. More specifically, rectangular openings are formed inthe exposure mask, and at the four corners of the opening, smallrectangular openings are formed in a protruding manner. When at leastone of the first filter, the second filter, the third filter, and thefourth filter is formed, the exposure mask with the corner serifpatterns is used. In the fifth embodiment, the resist material for anegative-type color filter is considered to be used.

More specifically, on the basis of a well-known method, photoelectricconversion elements 42, 52, 62, 72 are formed on a silicon semiconductorsubstrate 80, and subsequently, a first planarization film 82 and alight shield layer 86 are formed thereon. Subsequently, a resistmaterial including a pigment-based third material constituting the thirdfilter 63 is applied to the entire surface, and the resist material isexposed and developed using the exposure mask as shown in FIG. 12, sothat the third filter 63 of which four corners are swelling can beobtained as shown in FIG. 13A. Subsequently, a resist material includinga pigment-based second material constituting the second filter 53 isapplied to the entire surface, and the resist material is exposed anddeveloped using the exposure mask as shown in FIG. 12, so that thesecond filter 53 of which four corners are swelling can be obtained asshown in FIG. 13B. Subsequently, a resist material including apigment-based first material constituting the first filter 43 is appliedto the entire surface, and the resist material is exposed and developedusing the exposure mask as shown in FIG. 12, so that the first filter 43of which four corners are swelling can be obtained as shown in FIG. 14A.Subsequently, a resist material including the pigment-based firstmaterial, second material, and third material (or the pigment-basedfirst material and third material) constituting the fourth filter 73 isapplied to the entire surface, and the resist material is exposed anddeveloped using the exposure mask as shown in FIG. 12, so that thefourth filter 73 of which four corners are swelling can be obtained asshown in FIG. 14B. The four corners of each of the filters 43, 53, 63,73 are swelling, and in addition, the four corners of each of thefilters 43, 53, 63, 73 overlap each other, and therefore, there would beno gap between the filters 43, 53, 63, 73.

The method for forming the filters 43, 53, 63, 73 is not limited to theabove method. For example, a resist material including a pigment-basedsecond material constituting the second filter 53 is applied to theentire surface, and the resist material is exposed and developed usingthe exposure mask as shown in FIG. 12, so that the second filter 53 ofwhich four corners are swelling can be obtained as shown in FIG. 15A.Subsequently, a resist material including a pigment-based third materialconstituting the third filter 63 is applied to the entire surface, andthe resist material is exposed and developed using the exposure maskhaving a band-like opening, so that the third filter 63 can be obtainedas shown in FIG. 15B, and in addition, the third material layer can beformed in an area where the fourth filter 73 is to be formedsubsequently. Subsequently, a resist material including a pigment-basedfirst material constituting the first filter 43 is applied to the entiresurface, and the resist material is exposed and developed using theexposure mask having a band-like opening, so that the first filter 43can be obtained as shown in FIG. 16A, and in addition, the firstmaterial layer can be formed in an area where the fourth filter 73 is tobe formed on the band-like pattern 63 as shown in FIG. 15B. Thus, thefourth filter 73 can be obtained, which has two-layer stacked structureincluding the first material layer made of the first materialconstituting the first filter 43 and the third material layer made ofthe third material constituting the third filter 63. In this manner, bystacking the first filter and the third filter, the ND filter (fourthfilter 73) can be formed on the basis of the first material constitutingthe first filter which is magenta-like but is an existing material andthe third material constituting the third filter.

Alternatively, a resist material including a pigment-based secondmaterial constituting the second filter 53 is applied to the entiresurface, and the resist material is exposed and developed using theexposure mask, so that the second filter 53 of which four corners areswelling can be obtained as shown in FIG. 16B, and in addition, a layermade of the second material can be formed in an area where the fourthfilter 73 is to be formed subsequently. Thereafter, by forming the thirdfilter 63 based on formation of the layer made of the band-like thirdmaterial which is similar to what is shown in FIG. 15B and forming thefirst filter 43 based on formation of the layer made of the band-likefirst material which is similar to what is shown in FIG. 16A, the fourthfilter 73 can be obtained that has the three-layer stacked layerstructure including the first material layer made of the first materialconstituting the first filter 43, the second material layer made of thesecond material constituting the second filter 53, and the thirdmaterial layer made of the third material constituting the third filter63.

It should be noted that the order in which the first filter, the secondfilter, the third filter, and the fourth filter are formed is merely anexample, and may be changed as necessary, the order in which the firstmaterial layer, the second material layer, and the third material layerare stacked in the fourth filter is also merely an example, and may bechanged as necessary.

Sixth Embodiment

The sixth embodiment relates to an image-capturing device according tothe second aspect and the fourth aspect of the present disclosure. Aconceptual diagram of an image-capturing unit of the image-capturingdevice of the sixth embodiment is shown in FIG. 17A. Conceptual diagramsfor explaining operation of the image-capturing unit are shown in FIGS.18A, 18B, 19A, 19B, 20, and 21. Light reception characteristics of thefirst image-capturing element, the second image-capturing element, thethird image-capturing element, and the fourth image-capturing elementare schematically shown in FIG. 17B. It should be noted that theconceptual diagram of the image-capturing device and the schematicpartial cross sectional view of the image-capturing element schematicpartial cross sectional view are similar to what are shown in FIGS. 7Aand 7B.

The image-capturing device 2 of the sixth embodiment is animage-capturing device for obtaining a single-color image. When this isexplained according to the second aspect of the present disclosure, theimage-capturing device 2 of the sixth embodiment is an image-capturingdevice that includes an image-capturing unit 130 and an image processingunit 11, wherein the image-capturing unit 130 includes image-capturingelement units 131 arranged in a two-dimensional matrix form, and each ofthe image-capturing element units 131 includes: a first image-capturingelement 141 and a third image-capturing element 161 including a firstphotoelectric conversion element and configured to receive light in avisible light range, and a second image-capturing element 151 and afourth image-capturing element 171 including a neutral density filterand a second photoelectric conversion element and configured to receivelight in a visible light range.

Then, image processing unit 11 generates high sensitivity image data onthe basis of outputs from the first image-capturing element 141 and thethird image-capturing element 161, and generates low sensitivity imagedata on the basis of outputs from the second image-capturing element 151and the fourth image-capturing element 171. Further, the imageprocessing unit 11 generates a combined image using high sensitivityimage data corresponding to a low illumination image area in the lowillumination image area obtained from the low sensitivity image data orthe high sensitivity image data, and using low sensitivity image datacorresponding to a high illumination image area in the high illuminationimage area obtained from the low sensitivity image data or the highsensitivity image data.

The image-capturing device 2 of the sixth embodiment is animage-capturing device for obtaining a single-color image. When this isexplained according to the fourth aspect of the present disclosure, theimage-capturing device 2 of the sixth embodiment is an image-capturingdevice that includes an image-capturing unit and an image processingunit 11, wherein the image-capturing unit is arranged withimage-capturing element units, and each of the image-capturing elementunits includes: first image-capturing elements 141, 161 including afirst photoelectric conversion element and configured to receive lightin a visible light range and second image-capturing elements 151, 171including a neutral density filter and a second photoelectric conversionelement and configured to receive light in a visible light range.Although not shown in the drawing, the image-capturing device accordingto the fourth aspect is configured such that, for example, theimage-capturing element units are all arranged in a row of theimage-capturing unit along the first direction, or the image-capturingelement units are arranged, with an interval therebetween, in a row ofthe image-capturing unit along the first direction, and further, theimage-capturing element units are arranged at every M (however, M 2)rows along the second direction. A portion constituting theimage-capturing unit except the image-capturing element units may beoccupied by the first image-capturing elements.

Then, the image processing unit 11 generates high sensitivity image dataon the basis of outputs from the first image-capturing elements 141,161, and generates low sensitivity image data on the basis of outputsfrom the second image-capturing elements 151, 171. Further, the imageprocessing unit 11 generates a combined image using high sensitivityimage data corresponding to a low illumination image area in the lowillumination image area obtained from the low sensitivity image data orthe high sensitivity image data, and using low sensitivity image datacorresponding to a high illumination image area in the high illuminationimage area obtained from the low sensitivity image data or the highsensitivity image data.

In this case, the neutral density filter is made of a material obtainedby mixing a pigment-based first material constituting the first filter43, a pigment-based second material constituting the second filter 53,and a pigment-based third material constituting the third filter 63 ofthe first embodiment, or the fourth filter 73 is made of the three-layerstructure including the first material layer made of the pigment-basedfirst material constituting the first filter 43, the second materiallayer made of the pigment-based second material constituting the secondfilter 53, and the third material layer made of the pigment-based thirdmaterial constituting the third filter 63. By employing theconfiguration and the structure explained above and the neutral densityfilter where the film thickness of each material layer is setappropriately, the same spectral characteristic as the spectralcharacteristic of the filter obtained by stacking the first filter andthe third filter can be given to the neutral density filter.

Alternatively, the neutral density filter is made of a material obtainedby mixing the pigment-based second material constituting the secondfilter 53 according to the first embodiment and magenta material, or theneutral density filter 73 may be constituted by two-layer structureincluding the second material layer made of the pigment-based secondmaterial constituting the second filter 53 and the material layer madeof magenta material. In such configuration, the same spectralcharacteristic as the spectral characteristic of the filter obtained bystacking the first filter and the third filter can also be given to theneutral density filter where the film thickness of each material filteris set appropriately.

As shown in FIG. 17A, in the image-capturing device 2 of the sixthembodiment, the arrangement of the four image-capturing elements 141,151, 161, 171 (indicated by squares of solid lines and dotted lines inFIG. 17A) in any given image-capturing element unit 131 (indicated by asquare of solid lines in FIG. 17A) is the Bayer arrangement. Morespecifically, the image-capturing element units 131 are arranged in atwo-dimensional matrix form, and the image-capturing element units 131are arranged in the first direction and the second directionperpendicular to the first direction. In any given image-capturingelement unit 131, the third image-capturing element 161 and the fourthimage-capturing element 171 are arranged adjacently to each other alongthe first direction, the first image-capturing element 141 and thesecond image-capturing element 151 are arranged adjacently to each otheralong the first direction, the first image-capturing element 141 and thefourth image-capturing element 171 are arranged adjacently to each otheralong the second direction, and the second image-capturing element 151and the third image-capturing element 161 are arranged adjacently toeach other along the second direction. The sensitivities of the secondimage-capturing element 151 and the fourth image-capturing element 171are less than the sensitivities of the first image-capturing element 141and the third image-capturing element 161.

Then, the image processing unit 11 generates high sensitivity image dataon the basis of outputs from the first image-capturing element 141 andthe third image-capturing element 161, and generates low sensitivityimage data on the basis of outputs from the second image-capturingelement 151 and the fourth image-capturing element 171.

More specifically, when a certain image-capturing scene is captured, thefirst image-capturing element 141 and the third image-capturing element161, and the second image-capturing element 151 and the fourthimage-capturing element 171 output signal values “GY₁”, “GY₃₁”, “gy₂”,and “gy₄”, respectively, as shown in FIGS. 18A and 18B.

It should be noted that the light reception sensitivities of the firstphotoelectric conversion element constituting the first image-capturingelement 141 and the photoelectric conversion element constituting thethird image-capturing element 161 are both configured to be the same,“1”, for the sake of convenience and for simplifying the explanation,and the light reception sensitivities of the photoelectric conversionelements constituting the second image-capturing element 151 and thefourth image-capturing element 171 is “⅛” for the sake of convenience.This is considered to be also applicable to the following explanation.More specifically, a ratio between the average accumulationelectrical-charge quantity of the first image-capturing element 141 andthe third image-capturing element 161 and the accumulationelectrical-charge quantity of the second image-capturing element 151 andthe fourth image-capturing element 171 is 8:1 (18 dB). Due to thesensitivity difference of 18 dB, low sensitivity image data for threelevels of exposure are obtained. This is basically also applicable tothe following explanation.

Each image-capturing element having received the light in the state asshown in FIGS. 18A and 18B transmits the output (electric signal) basedon the received light quantity to the image processing unit 11. In theimage processing unit 11, for example, the interpolation processing isperformed on the basis of the following method.

More specifically, the first image-capturing element 141 is assumed tobe located in an area where the fourth image-capturing element 171 islocated, and an output that may be considered to be output by the firstimage-capturing element 141 that is assumed as above (assumed firstimage-capturing element 141) is interpolated from an output from a firstimage-capturing element 141 adjacent to the assumed firstimage-capturing element 141. For example, in FIG. 19A, an output thatmay be considered to be output by the (i+1, j+1)-th assumed firstimage-capturing element 141 is derived as an average value “GY′₄” ofoutputs from two image-capturing elements which are the (i+1, j)-thfirst image-capturing element 141 and the (i+1, j+1)-th firstimage-capturing element 141. The third image-capturing element 161 isassumed to be located in an area where the second image-capturingelement 151 is located, and an output that may be considered to beoutput by the third image-capturing element 161 that is assumed as above(assumed third image-capturing element 161) is interpolated from anoutput from a second image-capturing element 151 adjacent to the assumedthird image-capturing element 161. For example, in FIG. 19A, an outputthat may be considered to be output by the (i+1, j+1)-th assumed thirdimage-capturing element 161 is derived as an average value “GY′₂” ofoutputs from the two image-capturing elements which are the (i+1,j+1)-th third image-capturing element 161 and the (i+1, j+2)-th thirdimage-capturing element 161. In this manner, the same outputs (GY₁,GY′₂, GY₃, GY′₄) as the outputs from the image-capturing elements (onered color image-capturing element, two green color image-capturingelements, and one blue color image-capturing element) in an ordinaryimage-capturing device of the Bayer arrangement can be obtained (seeFIG. 19A), and the high sensitivity image data can be obtained from allof the obtained outputs.

The fourth image-capturing element 171 is assumed to be located in anarea where the first image-capturing element 141 is located, and anoutput that may be considered to be output by the fourth image-capturingelement 171 that is assumed as above (assumed fourth image-capturingelement 171) is interpolated from an output from a fourthimage-capturing element 171 adjacent to the assumed fourthimage-capturing element 171. For example, in FIG. 19B, an output thatmay be considered to be output by the (i+1, j+1)-th assumed fourthimage-capturing element 171 is derived as an average value “gy′₁” ofoutputs from two image-capturing elements which are the (i+1, j+1)-thfourth image-capturing element 171 and the (i+1, j+2)-th fourthimage-capturing element 171. The second image-capturing element 151 isassumed to be located in an area where the third image-capturing element161 is located, and an output that may be considered to be output by thesecond image-capturing element 151 that is assumed as above (assumedsecond image-capturing element 151) is interpolated from an output froma third image-capturing element 161 adjacent to the assumed secondimage-capturing element 151. For example, in FIG. 19B, an output thatmay be considered to be output by the (i+1, j+1)-th assumed secondimage-capturing element 151 is derived as an average value “gy′3” ofoutputs from two image-capturing elements which are the (i+1, j)-thsecond image-capturing element 151 and the (i+1, j+1)-th secondimage-capturing element 151. In this manner, the same outputs (gy′₁,gy₂, gy′₃, gy₄) as the outputs from the image-capturing elements (onered color image-capturing element, two green color image-capturingelements, and one blue color image-capturing element) in an ordinaryimage-capturing device of the Bayer arrangement can be obtained (seeFIG. 19B), and the low sensitivity image data can be obtained from allof the obtained outputs.

With such processing, the data configuration and the data structure ofthe low sensitivity image data and the data configuration and the datastructure of the high sensitivity image data can be configured to be thesame while the original number of pixels is not changed, and therefore,various kinds of signal processing can be simplified, and an image canbe generated that is combined by HDR imaging with a processing engine atan extremely low cost. Since the simplest periodicity property andsymmetric property are maintained as image processing, image qualitydegradation such as false color and jaggy is less likely to occur when acombined image is formed.

Subsequently, the image processing unit 11 generates a combined imageusing high sensitivity image data corresponding to a low illuminationimage area in the low illumination image area obtained from the lowsensitivity image data or the high sensitivity image data, and using lowsensitivity image data corresponding to a high illumination image areain the high illumination image area obtained from the low sensitivityimage data or the high sensitivity image data. More specifically, forexample, the low illumination image area and the high illumination imagearea are extracted from the high sensitivity image data. Morespecifically, in a case where the luminance average value in any givensize of area is equal to or more than a value defined in advance (forexample, equal to or more than the maximum illumination), this area maybe adopted as the high illumination image area, and in a case where theillumination average value is less than the value defined in advance,this area may be adopted as the low illumination image area. In FIG. 20,an area encircled by double lines with shade is considered to be thehigh illumination image area. Then, the combined image is generatedusing the high sensitivity image data corresponding to the lowillumination image area in the low illumination image area and using thelow sensitivity image data corresponding to the high illumination imagearea in the high illumination image area. This state is schematicallyshown in FIG. 21.

Seventh Embodiment

The seventh embodiment is a modification of the sixth embodiment, andrelates to an embodiment similar to the second embodiment. In theimage-capturing device of the seventh embodiment, a schematic partialtop view of a light shield layer 86 when the image-capturing element isconsidered to be cut along arrow B-B of FIG. 7B is shown in FIG. 22. Asshown in FIG. 22, a first aperture portion 147 is formed in a lightincidence area of the first photoelectric conversion element. A secondaperture portion 157 is formed between the neutral density filter andthe second photoelectric conversion element. A third aperture portion167 is formed in a light incidence area of the third photoelectricconversion element. A fourth aperture portion 177 is formed between theneutral density filter and the fourth photoelectric conversion element.The third aperture portion 167 is smaller than the first apertureportion 147, and the fourth aperture portion 177 is smaller than thesecond aperture portion 157.

For example, when the sizes of the first aperture portion 147, thesecond aperture portion 157, the third aperture portion 167, and thefourth aperture portion 177 are determined so that the light quantityattenuation for the second image-capturing element 151 and the fourthimage-capturing element 171 is twice as large as the light quantityattenuation for the first image-capturing element 141 and the thirdimage-capturing element 161, the sensitivity of the thirdimage-capturing element 161 is (½) of the sensitivity of the firstimage-capturing element 141, and the sensitivity of the fourthimage-capturing element 171 is ⅛ (the sixth embodiment) to 1/16 (24 dB)of the sensitivity of the second image-capturing element 151 (see FIGS.23A and 23B). Therefore, the high sensitivity image data [1] based onthe output from the first image-capturing element 141 and the highsensitivity image data [2] based on the output from the thirdimage-capturing element 161 can be obtained. However, a final outputratio of the high sensitivity image data [1] versus the high sensitivityimage data [2] is 1:(½) (see FIGS. 24A and 24B). The low sensitivityimage data [1] based on the output from the second image-capturingelement 151 and the low sensitivity image data [2] based on the outputfrom the fourth image-capturing element 171 can be obtained. However, afinal output ratio of the low sensitivity image data [1] versus the lowsensitivity image data [2] is (⅛):( 1/16) (see FIGS. 25A and 25B). Asdescribed above, in the seventh embodiment, four levels of informationcan be obtained by one image-capturing process. According to the seventhembodiment, the range from where information can be obtained becomes twotimes larger, and the lower limit of detection of a gray image isgreatly extended, and information can be obtained in a lowerillumination spatial area.

Eighth Embodiment

The eighth embodiment is also a modification of the sixth embodiment,and relates to an embodiment similar to the third embodiment. In theimage-capturing device of the eighth embodiment, an image processingunit 11 includes a first gain adjustment unit configured to adjustoutputs from the first image-capturing element 141 and the thirdimage-capturing element 161, and a second gain adjustment unitconfigured to adjust an output from the second image-capturing element151 and the fourth image-capturing element 171. Where an adjustmentcoefficient for the outputs from the first image-capturing element 141and the third image-capturing element 161 by the first gain adjustmentunit is denoted as Gn₁, and an adjustment coefficient for the outputfrom the second image-capturing element 151 and the fourthimage-capturing element 171 by the second gain adjustment unit isdenoted as Gn₂, Gn₁/Gn₂≧1 is satisfied, and more specifically, Gn₁/Gn₂=4is satisfied, although the present embodiment is not limited thereto.The range of gain difference of the first image-capturing element 141and the third image-capturing element 161 and the second image-capturingelement 151 and the fourth image-capturing element 171 is up to 12 dB.The sensitivity ratio of the first image-capturing element 141 and thethird image-capturing element 161 and the sensitivity ratio of thesecond image-capturing element 151 and the fourth image-capturingelement 171 is 1:(⅛). The gain ratio of the first image-capturingelement 141 and the third image-capturing element 161 and the secondimage-capturing element 151 and the fourth image-capturing element 171is 4:1. Therefore, the final output ratio of the first image-capturingelement 141 and the third image-capturing element 161 and the secondimage-capturing element 151 and the fourth image-capturing element 171is 1:( 1/32), that is, 30 dB.

The values of the adjustment coefficients Gn₁, Gn₂ may be fixed values,or may be automatically or manually changeable in accordance with theimage-capturing scene and the difference between the average luminancein the high sensitivity image data and the average luminance in the lowsensitivity image data, and therefore, image-capturing can be done withthe optimum dynamic range in accordance with various kinds of bright anddark scenes. The gain value for the first image-capturing element 141and the gain value for the third image-capturing element 161 may beconfigured to be different (for example, four times the gain value forthe first image-capturing element 141, and twice the gain value for thethird image-capturing element 161), and the gain value for the secondimage-capturing element 151 and the gain value for the fourthimage-capturing element 171 may be configured to be different (forexample, twice the gain value for the second image-capturing element151, and four times the gain value for the fourth image-capturingelement 171).

It should be noted that the seventh embodiment and the eighth embodimentmay be combined.

Ninth Embodiment

The ninth embodiment is also a modification of the sixth embodiment, andrelates to an embodiment similar to the fourth embodiment. Also in theninth embodiment, the image processing unit 11 may generate N sets ofhigh sensitivity image data and low sensitivity image data of whichexposure times are different, and the image processing unit 11 mayfurther divide an illumination area of an image obtained from the lowsensitivity image data or the high sensitivity image data into 2N levelsof image areas ranging from an image area of which illumination is thelowest to an image area of which illumination is the highest, and maygenerate a combined image using N sets of high sensitivity image datarespectively corresponding to N levels of low illumination image areasin the N levels of low illumination image areas ranging from the imagearea of which illumination is the lowest to the image area of whichillumination is the N-th lowest, and using N sets of low sensitivityimage data respectively corresponding to N levels of high illuminationimage areas in the N levels of high illumination image areas rangingfrom the image area of which illumination is the (N+1)-th lowest to theimage area of which illumination is the highest. In the ninthembodiment, N is 2, and there is such relationship that theimage-capturing time for obtaining the high sensitivity image data andthe low sensitivity image data of the first set is twice theimage-capturing time for obtaining the high sensitivity image data andthe low sensitivity image data of the second set.

Like the third embodiment, the image processing unit 11 divides anillumination area of an image obtained from the low sensitivity imagedata or the high sensitivity image data into 4 levels of areas rangingfrom an area of which illumination is the lowest to an area of whichillumination is the highest, and generate a combined image using twosets of high sensitivity image data respectively corresponding to twolevels of low illumination image areas, that is, high sensitivity imagedata obtained from the “high sensitivity 1” and the “high sensitivity 2”shown in FIG. 11, in two levels of low illumination image areas rangingfrom the area of which illumination is the lowest to the area of whichillumination is the second lowest, and using two sets of low sensitivityimage data respectively corresponding to two levels of high illuminationimage areas, that is, low sensitivity image data obtained from the “lowsensitivity 1” and the “low sensitivity 2” shown in FIG. 11, in twolevels of high illumination image areas ranging from the area of whichillumination is the third lowest to the fourth area of whichillumination is the highest. As a result, the dynamic range of the imagecombined by HDR imaging is 24 dB.

Also in the ninth embodiment, the range in which the color informationcan be obtained is twice as large as the conventional range, and thelower limit of detection of the gray image is also extended to the half,and therefore, information can be obtained in a lower illuminationspatial area. Therefore, as compared to the image combined by HDRimaging from an ordinary unit frame, not only the dynamic range isexpanded, but also information about intermediate illumination isincreased, and therefore, an image made by highly precise HDR imagingcan be formed from two unit frames.

It should be noted that the configuration and the structure of theimage-capturing device of the seventh embodiment can be applied to theimage-capturing device of the ninth embodiment. The configuration andthe structure of the image-capturing device of the eighth embodiment canbe applied to the image-capturing device of the ninth embodiment. Theconfiguration and the structure of the image-capturing device of theseventh embodiment and the eighth embodiment can be applied to theimage-capturing device of the ninth embodiment.

Tenth Embodiment

The tenth embodiment is a modification of the first embodiment and thesixth embodiment. More specifically, in the tenth embodiment, asingle-color image is obtained from the image-capturing device explainedin the first embodiment. In this case, the first gain adjustment unitprovided in the image processing unit 11 adjusts outputs from the firstimage-capturing element 41, the second image-capturing element 51, andthe third image-capturing element 61, so that the spectralcharacteristics based on the outputs become the same as the spectralcharacteristic of the fourth filter (neutral density filter) 73 of thefourth image-capturing element 71 as much as possible. For example, whenthe fourth filter is constituted by stacking the first filter and thethird filter as described above, the output of the first image-capturingelement 41 may be multiplied by 1.0, the output of the secondimage-capturing element 51 may be multiplied by 0.2, and the output ofthe third image-capturing element 61 may be multiplied by 1.0. In thiscase, these output multiplication factors are merely examples, and maybe configured appropriately so as to be similar to the spectralcharacteristic of the fourth filter. As explained in the sixthembodiment, the neutral density filter is made of a material obtained bymixing the first material, the second material, and the third material,or may be constituted by three-layer structure including the firstmaterial layer, the second material layer, and the third material layer,or may be made of a material obtained by mixing the first material andthe third material, or may be constituted by two-layer structureincluding the first material layer and the third material layer, or amaterial obtained by mixing the second material and magenta material, ormay be constituted by a two-layer structure including the secondmaterial layer and a material layer made of magenta material. It shouldbe noted that the spectral characteristic of a material obtained bymixing the first material and the third material is schematically shownin FIG. 17B.

Eleventh Embodiment

The eleventh embodiment relates to an image-capturing device accordingto the third aspect of the present disclosure. A conceptual diagram ofan image-capturing unit of the image-capturing device according to theeleventh embodiment is shown in FIG. 26. Conceptual diagrams forexplaining operation of the image-capturing unit are shown in FIGS. 27A,27B, 28A, 28B, and 29.

The image-capturing device of the eleventh embodiment is animage-capturing device for obtaining a combined image in which a portionof the image is a single-color image and the remaining portion thereofis a color image. The image-capturing device of the eleventh embodimentis an image-capturing device that includes an image-capturing unit 230and an image processing unit 11, wherein the image-capturing unit 230includes image-capturing element units 231A arranged in atwo-dimensional matrix form, and each of the image-capturing elementunits 231A includes four image-capturing element sub-units 231Bincluding a first image-capturing element 41 including a first filter 43and a first photoelectric conversion element 42 and configured toreceive light in a first wavelength band, a second image-capturingelement 51 including a second filter 53 and a second photoelectricconversion element 52 and configured to receive light in a secondwavelength band having a peak wavelength longer than a peak wavelengthof the first wavelength band, a third image-capturing element 61including a third filter 63 and a third photoelectric conversion element62 and configured to receive light in a third wavelength band having apeak wavelength longer than the peak wavelength of the second wavelengthband, and a fourth image-capturing element 71 including a fourth filter73 and a fourth photoelectric conversion element 72 and configured toreceive light in the first wavelength band, the second wavelength band,and the third wavelength band. The first image-capturing element 41, thesecond image-capturing element 51, the third image-capturing element 61,and the fourth image-capturing element 71, the photoelectric conversionelement, the filters 43, 53, 63, 73, and the like of the eleventhembodiment have the same configuration and the same structure as thoseof the first embodiment.

Like the first embodiment, the light transmission rate of the fourthfilter 73 of the image-capturing device of the eleventh embodiment isless than the light transmission rate of the first filter 43, the lighttransmission rate of the second filter 53, and the light transmissionrate of the third filter 63. Alternatively, the sensitivity of thefourth image-capturing element 71 is less than the sensitivities of thefirst image-capturing element 41, the second image-capturing element 51,and the third image-capturing element 61. The image processing unit 11generates high sensitivity image data on the basis of an outputsummation of the first image-capturing elements 41, an output summationof the second image-capturing elements 51, and an output summation ofthe third image-capturing elements 61 of the four image-capturingelement sub-units 231B constituting the image-capturing element unit231A, and generates low sensitivity image data on the basis of an outputfrom the fourth image-capturing elements 71 of each of the fourimage-capturing element sub-units 231B. More specifically, the totaloutput of the first image-capturing elements 41, the total output of thesecond image-capturing elements 51, and the total output of the thirdimage-capturing elements 61 are four times the output of each of thefourth image-capturing elements 71, and the sensitivity ratio of thesensitivity of the first image-capturing element 41, the secondimage-capturing element 51, and the third image-capturing element 61 andthe sensitivity of the fourth image-capturing element 71 is 1:(⅛), andtherefore, the final output ratio of the high sensitivity image data andthe low sensitivity image data is 1:( 1/32).

Further, the image processing unit 11 generates a combined image usinghigh sensitivity image data corresponding to a low illumination imagearea in the low illumination image area obtained from the lowsensitivity image data or the high sensitivity image data and using lowsensitivity image data corresponding to a high illumination image areain the high illumination image area obtained from the low sensitivityimage data or the high sensitivity image data.

It should be noted that the arrangement of the four image-capturingelements 41, 51, 61, 71 of any given image-capturing element sub-unit231B is the Bayer arrangement. In FIG. 26, the image-capturing elementunit 231A is indicated as being enclosed by solid lines, and theimage-capturing element sub-unit 231B is indicated as being enclosed bysolid lines and dotted lines. More specifically, for example, in anygiven image-capturing element sub-unit 231B, the third image-capturingelement 61 and the fourth image-capturing element 71 are arrangedadjacently to each other along the first direction, the firstimage-capturing element 41 and the second image-capturing element 51 arearranged adjacently to each other along the first direction, the firstimage-capturing element 41 and the fourth image-capturing element 71 arearranged adjacently to each other along the second direction, and thesecond image-capturing element 51 and the third image-capturing element61 are arranged adjacently to each other along the second direction.

When a certain image-capturing scene is captured, four firstimage-capturing elements 41 in any given image-capturing element unit231A output signal values “B¹”, “B²”, “B³”, “B⁴” as shown in FIG. 27A.Four second image-capturing elements 51 in any given image-capturingelement unit 231A output signal values “G₁ ¹”, “G₁ ²”, “G₁ ³”, “G₁ ⁴” asshown in FIG. 27A. Further, four third image-capturing elements 61 inany given image-capturing element unit 231A output signal values “R₁ ¹”,“R₁ ²”, “R₁ ³”, “R₁ ⁴” as shown in FIG. 27A. On the other hand, fourfourth image-capturing elements 71 in any given image-capturing elementunit 231A output signal values “Gy₄ ¹”, “Gy₄ ²”, “Gy₄ ³”, “Gy₄ ⁴” asshown in FIG. 27B.

Each image-capturing element having received light in the state as shownin FIGS. 27A and 27B transmits an output (electric signal) based on thereceived light quantity to the image processing unit 11. In the imageprocessing unit 11, for example, the interpolation processing isperformed on the basis of the following method.

More specifically, the image processing unit 11, derives outputsummations B¹⁻⁴, G₁ ¹⁻⁴, R¹⁻⁴ of a cluster of four first image-capturingelements 41 (first image-capturing element cluster), a cluster of foursecond image-capturing elements 51 (second image-capturing elementcluster), and a cluster of four third image-capturing elements 61 (thirdimage-capturing element cluster) of the four image-capturing elementsub-units 231B (see FIG. 28A). Further, the second image-capturingelement cluster is assumed to be located in an area where the fourthimage-capturing element cluster is located, and an output that may beconsidered to be output by the second image-capturing element cluster(assumed second image-capturing element cluster) that is assumed asabove is interpolated from an output from a second image-capturingelement cluster adjacent to the assumed second image-capturing elementcluster, so that an output G′₂ ¹⁻⁴ which is the same as the output froman image-capturing element in an ordinary image-capturing device of theBayer arrangement is obtained (see FIG. 28B). An output based on thefourth image-capturing element 71 similar to an output from animage-capturing element in an ordinary image-capturing device of theBayer arrangement is obtained without performing adding processing onthe basis of the fourth image-capturing element 71 constituting thefirst image-capturing element sub-unit 231B₁, the fourth image-capturingelement 71 constituting the second image-capturing element sub-unit231B₂, the fourth image-capturing element 71 constituting the thirdimage-capturing element sub-unit 231B₃, and the fourth image-capturingelement 71 constituting the fourth image-capturing element sub-unit231B₄ (see FIG. 29). Therefore, the high sensitivity image data (seeFIG. 28B) and the low sensitivity image data (see FIG. 29) of whichnumber of pixels is ¼ of the original number of pixels can be obtained,but the data configuration and the data structure of the low sensitivityimage data and the data configuration and the data structure of the highsensitivity image data can be configured to be the same, and therefore,various kinds of signal processing can be simplified. The highsensitivity image data are subjected to the four-pixel addingprocessing, but the low sensitivity image data are not subjected to theadding processing. Therefore, there is an advantage in that thesensitivity difference can be increased to four times.

Thereafter, by performing the same processing as the first embodiment onthe basis of the high sensitivity image data and the low sensitivityimage data, the image combined by HDR imaging can be obtained.

It should be noted that the configuration and the structure of theimage-capturing device explained in the second embodiment can also beapplied to the eleventh embodiment, the configuration and the structureof the image-capturing device explained in the third embodiment can alsobe applied to the eleventh embodiment, and the configuration and thestructure of the image-capturing device explained in the fourthembodiment can also be applied to the eleventh embodiment, or any givencombination thereof may also be applied to the eleventh embodiment.

The present disclosure has been hereinabove explained with reference tothe preferred embodiments, but the present disclosure is not limited tothese embodiments. The configuration and the structure of theimage-capturing device and the image-capturing element explained in theembodiments are merely examples, and can be changed as necessary. Forexample, image-capturing element may be configured such that aphotoelectric conversion element is provided on a silicon semiconductorsubstrate, and then a first planarization film, an on-chip lens, asecond planarization film, and a filter are stacked thereon. Theimage-capturing element may be a front-illuminated element instead of aback-illuminated element.

It should be noted that the present disclosure may be configured asfollows.

[1]<<Image-Capturing Device: First Aspect>>

An image-capturing device including: an image-capturing unit; and animage processing unit, wherein the image-capturing unit includesimage-capturing element units arranged in a two-dimensional matrix form,and each of the image-capturing element units includes: a firstimage-capturing element including a first filter and a firstphotoelectric conversion element, and configured to receive light in afirst wavelength band; a second image-capturing element including asecond filter and a second photoelectric conversion element, andconfigured to receive light in a second wavelength band having a peakwavelength longer than a peak wavelength of the first wavelength band; athird image-capturing element including a third filter and a thirdphotoelectric conversion element, and configured to receive light in athird wavelength band having a peak wavelength longer than a peakwavelength of the second wavelength band; and a fourth image-capturingelement including a fourth filter and a fourth photoelectric conversionelement, and configured to receive light in the first wavelength band,the second wavelength band, and the third wavelength band, and wherein alight transmission rate of the fourth filter is less than a lighttransmission rate of the first filter, a light transmission rate of thesecond filter, and a light transmission rate of the third filter, andthe image processing unit generates high sensitivity image data inaccordance with outputs from the first image-capturing element, thesecond image-capturing element, and the third image-capturing element,and generates low sensitivity image data in accordance with an outputfrom the fourth image-capturing element, and the image processing unitfurther generates a combined image using high sensitivity image datacorresponding to a low illumination image area in the low illuminationimage area obtained from the low sensitivity image data or the highsensitivity image data, and using low sensitivity image datacorresponding to an high illumination image area in the highillumination image area obtained from the low sensitivity image data orthe high sensitivity image data.

[2] The image-capturing device according to [1], wherein a firstaperture portion is formed between the first filter and the firstphotoelectric conversion element, a second aperture portion is formedbetween the second filter and the second photoelectric conversionelement, a third aperture portion is formed between the third filter andthe third photoelectric conversion element, and a fourth apertureportion is formed between the fourth filter and the fourth photoelectricconversion element, and wherein the fourth aperture portion is smallerthan the first aperture portion, the second aperture portion, and thethird aperture portion.[3] The image-capturing device according to [1] or [2], wherein theimage processing unit includes a first gain adjustment unit configuredto adjust outputs from the first image-capturing element, the secondimage-capturing element, and the third image-capturing element, and asecond gain adjustment unit configured to adjust an output from thefourth image-capturing element.[4] The image-capturing device according to [3], wherein where anadjustment coefficient for outputs from the first image-capturingelement, the second image-capturing element, and the thirdimage-capturing element by the first gain adjustment unit is denoted asGn₁, and an adjustment coefficient for an output from the fourthimage-capturing element by the second gain adjustment unit is denoted asGn₂, Gn₁/Gn₂≧1 is satisfied.[5] The image-capturing device according to any one of [1] to [4],wherein the image processing unit generates N sets of high sensitivityimage data and low sensitivity image data of which exposure times aredifferent, and the image processing unit further divides an illuminationarea of an image obtained from the low sensitivity image data or thehigh sensitivity image data into 2N levels of areas ranging from an areaof which illumination is the lowest to an area of which illumination isthe highest, and generates a combined image using N sets of highsensitivity image data respectively corresponding to N levels of lowillumination image areas in the N levels of low illumination image areasranging from the area of which illumination is the lowest to the area ofwhich illumination is the N-th lowest, and using N sets of lowsensitivity image data respectively corresponding to N levels of highillumination image areas in the N levels of high illumination imageareas ranging from the area of which illumination is the (N+1)-th lowestto the area of which illumination is the highest.[6] The image-capturing device according to [5], wherein N is 2, andthere is such relationship that the image-capturing time for obtainingthe high sensitivity image data and the low sensitivity image data ofthe first set is twice the image-capturing time for obtaining the highsensitivity image data and the low sensitivity image data of the secondset.[7] The image-capturing device according to any one of [1] to [6],wherein there is no gap between the first filter, the second filter, thethird filter, and the fourth filter.[8] The image-capturing device according to any one of [1] to [7],wherein the fourth filter has a three-layer stacked layer structureincluding a first material layer made of a first material constitutingthe first filter, a second material layer made of a second materialconstituting the second filter, and a third material layer made of athird material constituting the third filter.[9] The image-capturing device according to any one of [1] to [7],wherein the fourth filter is made of a material obtained by mixing afirst material constituting the first filter, a second materialconstituting the second filter, and a third material constituting thethird filter.[10] The image-capturing device according to any one of [1] to [7],wherein the fourth filter has a two layer stacked layer structureincluding a first material layer made of a first material constitutingthe first filter and a third material layer made of a third materialconstituting the third filter.[11] The image-capturing device according to any one of [1] to [7],wherein the fourth filter is made of a material obtained by mixing afirst material constituting the first filter and a third materialconstituting the third filter.

[12]<<Image-Capturing Device: Second Aspect>>

An image-capturing device including: an image-capturing unit; and animage processing unit, wherein the image-capturing unit includesimage-capturing element units arranged in a two-dimensional matrix form,and each of the image-capturing element units includes: a firstimage-capturing element and a third image-capturing element including afirst photoelectric conversion element and configured to receive lightin a visible light range; and a second image-capturing element and afourth image-capturing element including a neutral density filter and asecond photoelectric conversion element and configured to receive lightin a visible light range, wherein the image processing unit generateshigh sensitivity image data on the basis of outputs from the firstimage-capturing element and the third image-capturing element, andgenerates low sensitivity image data on the basis of outputs from thesecond image-capturing element and the fourth image-capturing element,and the image processing unit further generates a combined image usinghigh sensitivity image data corresponding to a low illumination imagearea in the low illumination image area obtained from the lowsensitivity image data or the high sensitivity image data and using lowsensitivity image data corresponding to a high illumination image areain the high illumination image area obtained from the low sensitivityimage data or the high sensitivity image data.

[13] The image-capturing device according to [12], wherein a firstaperture portion is formed in a light incidence area of the firstphotoelectric conversion element, a second aperture portion is formedbetween the neutral density filter and the second photoelectricconversion element, a third aperture portion is formed in a lightincidence area of the third photoelectric conversion element, and afourth aperture portion is formed between the neutral density filter andthe fourth photoelectric conversion element, wherein the third apertureportion is smaller than the first aperture portion, and the fourthaperture portion is smaller than the second aperture portion.[14] The image-capturing device according to [12] or [13], wherein theimage processing unit has a first gain adjustment unit configured toadjust outputs from the first image-capturing element and the thirdimage-capturing element, and an second gain adjustment unit configuredto adjust outputs from the second image-capturing element and the fourthimage-capturing element.[15] The image-capturing device according to [14], wherein where anadjustment coefficient for outputs from the first image-capturingelement and the third image-capturing element by the first gainadjustment unit is denoted as Gn₁, and an adjustment coefficient foroutputs from the second image-capturing element and the fourthimage-capturing element by the second gain adjustment unit is denoted asGn₂, Gn₁/Gn₂≧1 is satisfied.[16] The image-capturing device according to any one of [12] to [15],wherein the image processing unit generates N sets of high sensitivityimage data and low sensitivity image data of which exposure times aredifferent, and the image processing unit further divides an illuminationarea of an image obtained from the low sensitivity image data or thehigh sensitivity image data into 2N levels of image areas ranging froman image area of which illumination is the lowest to an image area ofwhich illumination is the highest, and generates a combined image usingN sets of high sensitivity image data respectively corresponding to Nlevels of low illumination image areas in the N levels of lowillumination image areas ranging from the image area of whichillumination is the lowest to the image area of which illumination isthe N-th lowest, and using N sets of low sensitivity image datarespectively corresponding to N levels of high illumination image areasin the N levels of high illumination image areas ranging from the imagearea of which illumination is the (N+1)-th lowest to the image area ofwhich illumination is the highest.[17] The image-capturing device according to [16], wherein N is 2, andthere is such relationship that the image-capturing time for obtainingthe high sensitivity image data and the low sensitivity image data ofthe first set is twice the image-capturing time for obtaining the highsensitivity image data and the low sensitivity image data of the secondset.[18] The image-capturing device according to any one of [12] to [17],wherein the neutral density filter has the same spectral characteristicas a spectral characteristic of a filter made by stacking the firstfilter and the third filter.

[19]<<Image-Capturing Device: Third Aspect>>

An image-capturing device including: an image-capturing unit; and animage processing unit, wherein the image-capturing unit includesimage-capturing element units arranged in a two-dimensional matrix form,and each of the image-capturing element units includes fourimage-capturing element sub-units each including a first image-capturingelement including a first filter and a first photoelectric conversionelement and configured to receive light in a first wavelength band; asecond image-capturing element including a second filter and a secondphotoelectric conversion element and configured to receive light in asecond wavelength band having a peak wavelength longer than a peakwavelength of the first wavelength band; a third image-capturing elementincluding a third filter and a third photoelectric conversion elementand configured to receive light in a third wavelength band having a peakwavelength longer than a peak wavelength of the second wavelength band;and a fourth image-capturing element including a fourth filter and afourth photoelectric conversion element and configured to receive lightin the first wavelength band, the second wavelength band, and the thirdwavelength band, wherein a light transmission rate of the fourth filteris less than a light transmission rate of the first filter, a lighttransmission rate of the second filter, and a light transmission rate ofthe third filter, and the image processing unit generates highsensitivity image data on the basis of an output summation of the firstimage-capturing elements, an output summation of the secondimage-capturing elements, and an output summation of the thirdimage-capturing elements of the four image-capturing element sub-unitsconstituting the image-capturing element unit, and generates lowsensitivity image data on the basis of an output from the fourthimage-capturing element of each of the four image-capturing elementsub-units, and the image processing unit further generates a combinedimage using high sensitivity image data corresponding to a lowillumination image area in the low illumination image area obtained fromthe low sensitivity image data or the high sensitivity image data, andusing low sensitivity image data corresponding to an high illuminationimage area in the high illumination image area obtained from the lowsensitivity image data or the high sensitivity image data.

[20]<<Image-Capturing Device: Fourth Aspect>>

An image-capturing device including: an image-capturing unit; and animage processing unit, wherein the image-capturing unit is arranged withimage-capturing element units, each of the image-capturing element unitsincludes: a first image-capturing element including a firstphotoelectric conversion element and configured to receive light in avisible light range; and a second image-capturing element including aneutral density filter and a second photoelectric conversion element andconfigured to receive light in a visible light range, and the imageprocessing unit generates high sensitivity image data on the basis of anoutput from the first image-capturing element and generates lowsensitivity image data on the basis of an output from the secondimage-capturing element, and the image processing unit further generatesa combined image using high sensitivity image data corresponding to alow illumination image area in the low illumination image area obtainedfrom the low sensitivity image data or the high sensitivity image dataand using low sensitivity image data corresponding to a highillumination image area in the high illumination image area obtainedfrom the low sensitivity image data or the high sensitivity image data.

REFERENCE SIGNS LIST

-   1, 2 image-capturing device-   2 camera main body-   11 image processing unit-   12 image storage unit-   20 lens system-   21 image-capturing lens-   22 diaphragm unit-   23 imaging lens-   30, 130, 230 image-capturing unit-   31, 131, 231A image-capturing element unit-   231B image-capturing element sub-unit-   41, 141 first image-capturing element-   42 first photoelectric conversion element-   43 first filter-   51, 151 second image-capturing element-   52 second photoelectric conversion element-   53 second filter-   61, 161 third image-capturing element-   62 third photoelectric conversion element-   63 third filter-   71, 171 fourth image-capturing element-   72 fourth photoelectric conversion element-   73 fourth filter-   80 silicon semiconductor substrate-   82 first planarization film-   84 on-chip lens-   85 second planarization film

1. An image-capturing device comprising: an image-capturing unit; and animage processing unit, wherein the image-capturing unit includesimage-capturing element units arranged in a two-dimensional matrix form,and each of the image-capturing element units includes: a firstimage-capturing element including a first filter and a firstphotoelectric conversion element, and configured to receive light in afirst wavelength band; a second image-capturing element including asecond filter and a second photoelectric conversion element, andconfigured to receive light in a second wavelength band having a peakwavelength longer than a peak wavelength of the first wavelength band; athird image-capturing element including a third filter and a thirdphotoelectric conversion element, and configured to receive light in athird wavelength band having a peak wavelength longer than a peakwavelength of the second wavelength band; and a fourth image-capturingelement including a fourth filter and a fourth photoelectric conversionelement, and configured to receive light in the first wavelength band,the second wavelength band, and the third wavelength band, and wherein alight transmission rate of the fourth filter is less than a lighttransmission rate of the first filter, a light transmission rate of thesecond filter, and a light transmission rate of the third filter, andthe image processing unit generates high sensitivity image data inaccordance with outputs from the first image-capturing element, thesecond image-capturing element, and the third image-capturing element,and generates low sensitivity image data in accordance with an outputfrom the fourth image-capturing element, and the image processing unitfurther generates a combined image using high sensitivity image datacorresponding to a low illumination image area in the low illuminationimage area obtained from the low sensitivity image data or the highsensitivity image data, and using low sensitivity image datacorresponding to an high illumination image area in the highillumination image area obtained from the low sensitivity image data orthe high sensitivity image data.
 2. The image-capturing device accordingto claim 1, wherein a first aperture portion is formed between the firstfilter and the first photoelectric conversion element, a second apertureportion is formed between the second filter and the second photoelectricconversion element, a third aperture portion is formed between the thirdfilter and the third photoelectric conversion element, and a fourthaperture portion is formed between the fourth filter and the fourthphotoelectric conversion element, and wherein the fourth apertureportion is smaller than the first aperture portion, the second apertureportion, and the third aperture portion.
 3. The image-capturing deviceaccording to claim 1, wherein the image processing unit includes a firstgain adjustment unit configured to adjust outputs from the firstimage-capturing element, the second image-capturing element, and thethird image-capturing element, and a second gain adjustment unitconfigured to adjust an output from the fourth image-capturing element.4. The image-capturing device according to claim 3, wherein where anadjustment coefficient for an output from the first image-capturingelement, the second image-capturing element, and the thirdimage-capturing element by the first gain adjustment unit is denoted asGn₁, and an adjustment coefficient for an output from the fourthimage-capturing element by the second gain adjustment unit is denoted asGn₂, Gn₁/Gn₂≧1 is satisfied.
 5. The image-capturing device according toclaim 1, wherein the image processing unit generates N sets of highsensitivity image data and low sensitivity image data of which exposuretimes are different, and the image processing unit further divides anillumination area of an image obtained from the low sensitivity imagedata or the high sensitivity image data into 2N levels of areas rangingfrom an area of which illumination is the lowest to an area of whichillumination is the highest, and generates a combined image using N setsof high sensitivity image data respectively corresponding to N levels oflow illumination image areas in the N levels of low illumination imageareas ranging from the area of which illumination is the lowest to thearea of which illumination is the N-th lowest, and using N sets of lowsensitivity image data respectively corresponding to N levels of highillumination image areas in the N levels of high illumination imageareas ranging from the area of which illumination is the (N+1)-th lowestto the area of which illumination is the highest.
 6. The image-capturingdevice according to claim 5, wherein N is 2, and there is suchrelationship that the image-capturing time for obtaining the highsensitivity image data and the low sensitivity image data of the firstset is twice the image-capturing time for obtaining the high sensitivityimage data and the low sensitivity image data of the second set.
 7. Theimage-capturing device according to claim 1, wherein there is no gapbetween the first filter, the second filter, the third filter, and thefourth filter.
 8. The image-capturing device according to claim 1,wherein the fourth filter has a three-layer stacked layer structureincluding a first material layer made of a first material constitutingthe first filter, a second material layer made of a second materialconstituting the second filter, and a third material layer made of athird material constituting the third filter.
 9. The image-capturingdevice according to claim 1, wherein the fourth filter is made of amaterial obtained by mixing a first material constituting the firstfilter, a second material constituting the second filter, and a thirdmaterial constituting the third filter
 10. The image-capturing deviceaccording to claim 1, wherein the fourth filter has a two layer stackedlayer structure including a first material layer made of a firstmaterial constituting the first filter and a third material layer madeof a third material constituting the third filter.
 11. Theimage-capturing device according to claim 1, wherein the fourth filteris made of a material obtained by mixing a first material constitutingthe first filter and a third material constituting the third filter. 12.An image-capturing device comprising: an image-capturing unit; and animage processing unit, wherein the image-capturing unit includesimage-capturing element units arranged in a two-dimensional matrix form,and each of the image-capturing element units includes: a firstimage-capturing element and a third image-capturing element including afirst photoelectric conversion element and configured to receive lightin a visible light range; and a second image-capturing element and afourth image-capturing element including a neutral density filter and asecond photoelectric conversion element and configured to receive lightin a visible light range, wherein the image processing unit generateshigh sensitivity image data on the basis of outputs from the firstimage-capturing element and the third image-capturing element, andgenerates low sensitivity image data on the basis of outputs from thesecond image-capturing element and the fourth image-capturing element,and the image processing unit further generates a combined image usinghigh sensitivity image data corresponding to a low illumination imagearea in the low illumination image area obtained from the lowsensitivity image data or the high sensitivity image data and using lowsensitivity image data corresponding to a high illumination image areain the high illumination image area obtained from the low sensitivityimage data or the high sensitivity image data.
 13. The image-capturingdevice according to claim 12, wherein a first aperture portion is formedin a light incidence area of the first photoelectric conversion element,a second aperture portion is formed between the neutral density filterand the second photoelectric conversion element, a third apertureportion is formed in a light incidence area of the third photoelectricconversion element, and a fourth aperture portion is formed between theneutral density filter and the fourth photoelectric conversion element,wherein the third aperture portion is smaller than the first apertureportion, and the fourth aperture portion is smaller than the secondaperture portion.
 14. The image-capturing device according to claim 12,wherein the image processing unit has a first gain adjustment unitconfigured to adjust outputs from the first image-capturing element andthe third image-capturing element, and an second gain adjustment unitconfigured to adjust outputs from the second image-capturing element andthe fourth image-capturing element.
 15. The image-capturing deviceaccording to claim 14, wherein where an adjustment coefficient foroutputs from the first image-capturing element and the thirdimage-capturing element by the first gain adjustment unit is denoted asGn₁, and an adjustment coefficient for outputs from the secondimage-capturing element and the fourth image-capturing element by thesecond gain adjustment unit is denoted as Gn₂, Gn₁/Gn₂≧1 is satisfied.16. The image-capturing device according to claim 12, wherein the imageprocessing unit generates N sets of high sensitivity image data and lowsensitivity image data of which exposure times are different, and theimage processing unit further divides an illumination area of an imageobtained from the low sensitivity image data or the high sensitivityimage data into 2N levels of image areas ranging from an image area ofwhich illumination is the lowest to an image area of which illuminationis the highest, and generates a combined image using N sets of highsensitivity image data respectively corresponding to N levels of lowillumination image areas in the N levels of low illumination image areasranging from the image area of which illumination is the lowest to theimage area of which illumination is the N-th lowest, and using N sets oflow sensitivity image data respectively corresponding to N levels ofhigh illumination image areas in the N levels of high illumination imageareas ranging from the image area of which illumination is the (N+1)-thlowest to the image area of which illumination is the highest.
 17. Theimage-capturing device according to claim 16, wherein N is 2, and thereis such relationship that the image-capturing time for obtaining thehigh sensitivity image data and the low sensitivity image data of thefirst set is twice the image-capturing time for obtaining the highsensitivity image data and the low sensitivity image data of the secondset.
 18. The image-capturing device according to claim 12, wherein theneutral density filter has the same spectral characteristic as aspectral characteristic of a filter made by stacking the first filterand the third filter.
 19. An image-capturing device comprising: animage-capturing unit; and an image processing unit, wherein theimage-capturing unit includes image-capturing element units arranged ina two-dimensional matrix form, and each of the image-capturing elementunits includes four image-capturing element sub-units each including afirst image-capturing element including a first filter and a firstphotoelectric conversion element and configured to receive light in afirst wavelength band; a second image-capturing element including asecond filter and a second photoelectric conversion element andconfigured to receive light in a second wavelength band having a peakwavelength longer than a peak wavelength of the first wavelength band; athird image-capturing element including a third filter and a thirdphotoelectric conversion element and configured to receive light in athird wavelength band having a peak wavelength longer than a peakwavelength of the second wavelength band; and a fourth image-capturingelement including a fourth filter and a fourth photoelectric conversionelement and configured to receive light in the first wavelength band,the second wavelength band, and the third wavelength band, wherein alight transmission rate of the fourth filter is less than a lighttransmission rate of the first filter, a light transmission rate of thesecond filter, and a light transmission rate of the third filter, andthe image processing unit generates high sensitivity image data on thebasis of an output summation of the first image-capturing elements, anoutput summation of the second image-capturing elements, and an outputsummation of the third image-capturing elements of the fourimage-capturing element sub-units constituting the image-capturingelement unit, and generates low sensitivity image data on the basis ofan output from the fourth image-capturing element of each of the fourimage-capturing element sub-units, and the image processing unit furthergenerates a combined image using high sensitivity image datacorresponding to a low illumination image area in the low illuminationimage area obtained from the low sensitivity image data or the highsensitivity image data, and using low sensitivity image datacorresponding to an high illumination image area in the highillumination image area obtained from the low sensitivity image data orthe high sensitivity image data.
 20. An image-capturing devicecomprising: an image-capturing unit; and an image processing unit,wherein the image-capturing unit is arranged with image-capturingelement units, each of the image-capturing element units includes: afirst image-capturing element including a first photoelectric conversionelement and configured to receive light in a visible light range; and asecond image-capturing element including a neutral density filter and asecond photoelectric conversion element and configured to receive lightin a visible light range, and the image processing unit generates highsensitivity image data on the basis of an output from the firstimage-capturing element and generates low sensitivity image data on thebasis of an output from the second image-capturing element, and theimage processing unit further generates a combined image using highsensitivity image data corresponding to a low illumination image area inthe low illumination image area obtained from the low sensitivity imagedata or the high sensitivity image data and using low sensitivity imagedata corresponding to a high illumination image area in the highillumination image area obtained from the low sensitivity image data orthe high sensitivity image data.